-- Logs begin at Thu 2022-07-14 11:24:19 UTC, end at Thu 2022-07-14 12:37:54 UTC. -- Jul 14 11:58:20.329955 dv-cinder-848900-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 14 11:58:20.339492 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Jul 14 11:58:20 2022] *** Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: nodename: dv-cinder-848900-3-iscsi Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: machine: x86_64 Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: clock source: unix Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: pcre jit disabled Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: detected number of CPU cores: 4 Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: current working directory: / Jul 14 11:58:20.341523 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: detected binary path: /usr/bin/uwsgi-core Jul 14 11:58:20.342190 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: your processes number limit is 47680 Jul 14 11:58:20.342190 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: your memory page size is 4096 bytes Jul 14 11:58:20.342190 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: detected max file descriptor number: 2048 Jul 14 11:58:20.342190 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: lock engine: pthread robust mutexes Jul 14 11:58:20.342190 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: thunder lock: enabled Jul 14 11:58:20.342190 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 14 11:58:20.342190 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0] Jul 14 11:58:20.385426 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: Python main interpreter initialized at 0x56478e5c39c0 Jul 14 11:58:20.385426 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: python threads support enabled Jul 14 11:58:20.385426 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: your server socket listen backlog is limited to 100 connections Jul 14 11:58:20.385426 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: your mercy for graceful operations on workers is 90 seconds Jul 14 11:58:20.385806 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: mapped 403077 bytes (393 KB) for 2 cores Jul 14 11:58:20.385877 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: *** Operational MODE: preforking *** Jul 14 11:58:20.386149 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: *** uWSGI is running in multiple interpreter mode *** Jul 14 11:58:20.386149 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: spawned uWSGI master process (pid: 80486) Jul 14 11:58:20.386356 dv-cinder-848900-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 14 11:58:20.387769 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: spawned uWSGI worker 1 (pid: 80487, cores: 1) Jul 14 11:58:20.387769 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: spawned uWSGI worker 2 (pid: 80488, cores: 1) Jul 14 11:58:20.387769 dv-cinder-848900-3-iscsi devstack@keystone.service[80486]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 14 11:58:25.857181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 14 11:58:25.857181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: from cryptography.utils import int_from_bytes Jul 14 11:58:25.857951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 14 11:58:25.857951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: from cryptography.utils import int_from_bytes Jul 14 11:58:25.858391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 14 11:58:25.858391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: from cryptography.utils import int_from_bytes Jul 14 11:58:25.858525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 14 11:58:25.858525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: from cryptography.utils import int_from_bytes Jul 14 11:58:25.906056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 14 11:58:25.906241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 14 11:58:25.906368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 14 11:58:25.906472 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] command line args: [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 14 11:58:25.906554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 14 11:58:25.906676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 14 11:58:25.906756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] command line args: [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 14 11:58:25.906856 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ================================================================================ {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 14 11:58:25.907034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 14 11:58:25.907125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] admin_token = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.907212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ================================================================================ {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 14 11:58:25.907341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] config_dir = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.907458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] admin_token = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.907545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.907692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] config_source = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.907770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] config_dir = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.907870 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] debug = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.907956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] config_source = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] debug = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.908902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] list_limit = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] log_config_append = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] list_limit = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] log_config_append = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.909935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] log_dir = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] log_file = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] log_dir = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] log_file = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.910940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.911916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.912972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.913066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.913759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.913855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.914012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.914328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.914412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.914498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] publish_errors = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.914733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.914885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.27/identity {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.914992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] publish_errors = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915393 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] standard_threads = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.915997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] standard_threads = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] use_journal = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916817 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] use_json = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.916983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] use_stderr = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] use_journal = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] use_syslog = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917322 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] use_json = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917502 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] use_stderr = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.917730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] use_syslog = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.917891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 14 11:58:25.918144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.918969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.919934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.920909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.921881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.922881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.923999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.924948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.925971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.926128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.926212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.926377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.926463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.926608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.926687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.927957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.928085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.928164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.928297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.928415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.928492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.928608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.928693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.929484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.929484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.929630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.929741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.929830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.929980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.930216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.930302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.930457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.930535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.930687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.930823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.930948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.931971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.932919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.933862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.934999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.935078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.935242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.935326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.935478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.935847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.935919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.935979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.936966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.937988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.938971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.939912 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.940997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.941926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.942999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] auth.external = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] auth.external = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.943986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.944126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] auth.password = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.944217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.944357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] auth.token = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.944473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.944714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.944918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] auth.password = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] auth.token = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.945882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] credential.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.946980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] credential.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.947961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.948928 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.949943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.950940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.951894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.952865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.953985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.954948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.955993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.956929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.957903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.958876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.959940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.960514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.960730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.960878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.961979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.962943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.963884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.964842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.965939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.966990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967856 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.967936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.968864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.969976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.970942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.971963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.972915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.973845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974820 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.974982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.975077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.975218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.975299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.975456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.975547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.975766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.975853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.976925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] role.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] role.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.977957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] role.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] role.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.978909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.979872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.980962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.981201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.981327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.981454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.981587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.981714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.981842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.981969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.982105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.982239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.982365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.982491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.982617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.982743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.982924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.983966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.984986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.985116 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.985241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.985376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.985501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.985642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.985768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.985893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.986951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.987980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.988994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.989150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.989282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.989410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.989536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.989661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.989787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.989913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.connection = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.990994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.991122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.991249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.991824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.991933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.992062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.992197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.992449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.992583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.992712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.connection = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.992850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.992850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.993100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.993100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.993358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.993486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.993612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.993739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.993866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.994999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.995140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.995265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.995390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.995514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.995646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.995771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.995896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80487) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 14 11:58:25.996253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 14 11:58:25.996253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80488) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 14 11:58:26.025660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.025876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.026009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.026136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.026262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.026411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.034433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.034571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.063501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.063729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.064264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.064494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.205883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.207067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.255575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.255760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.255910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.256070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.321622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.321856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.322032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.322181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.322314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.322464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.322592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.322720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.322720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.323000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.323000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.323256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.347449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.347592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.347721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.347860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.347989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.348925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.360092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.360293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.373532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.373726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.373865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.373997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.374132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.374277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.374447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.374586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.374717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.374863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.374990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.375114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.375249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80488) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.375374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80487) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 14 11:58:26.391836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.392048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 14 11:58:26.392297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.392506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 14 11:58:26.392648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.392802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 14 11:58:26.393060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.393186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.393310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 14 11:58:26.393444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.393569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.393716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.393798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.393866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.393989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.394237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.404881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.405158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.405242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.405425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.405520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.405703 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.405784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.405995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.414456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.414811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.417740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.418022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.418105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.418306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.418386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.418581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.418661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.418887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.418968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.419150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.419243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.419408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.419489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.419668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.419749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.419932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.420011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.420194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.420271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.420457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.420543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.420719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.447354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.447722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.448943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.449221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.449323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.449582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.454465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.454747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.454854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.455046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.455132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.455327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.462479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.462784 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.463014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.463292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.467234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.467519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.467704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.467786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.467985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.468070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.468264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.468346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.468542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.468639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.468814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.468897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.469082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.469164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.469346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.469613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.488482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.488757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.488840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.489054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.494123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.494237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.497768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.497855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.498062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.498236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.503207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.503295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.506842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.507019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.510479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.510611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.510789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.510927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.511092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.511216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.511379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.511502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.511663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.511788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.511948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.512069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.531166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.531281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.531451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.531540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.531717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.531802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.531973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.532056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.540343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.540497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.542021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.542203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.542302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.542501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.545447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.545673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.545760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.545953 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.546040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.546234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.546319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.546515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.546597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.546823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.560333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.560608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.560857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.560940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.561137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.561235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.561418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.561502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.561692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.561776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.561965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.562051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.562244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.562517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.581028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.581751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.584287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.585074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.587628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.588384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.589201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.589978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.590982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.591262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.591547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.591755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.592034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.592324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.598723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.599606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.600338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.603008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.604076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.604363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.604643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.604914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.606227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.606517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.606820 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.607097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.614028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.614311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.614589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.616252 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.617153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.617438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.622938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80487) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.623217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.623496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.623776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.624042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.624324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.624596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.624865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.625230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.625311 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80488) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 14 11:58:26.625498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.625590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.625774 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.625858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.626043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.626125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.626305 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.626390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.626567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80487) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.626649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.626866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.627141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.627513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.627787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.628059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.628336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.628601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.628869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80488) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 14 11:58:26.664888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x56478e5c39c0 pid: 80487 (default app) Jul 14 11:58:26.667537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x56478e5c39c0 pid: 80488 (default app) Jul 14 11:58:26.668363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3377c-5414-4fdd-a1c6-6beaadc50173 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:26.668652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3377c-5414-4fdd-a1c6-6beaadc50173 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:26.668932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf3377c-5414-4fdd-a1c6-6beaadc50173 None None] PATH_INFO: `/v3` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:26.673180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1/1] 10.222.0.27 () {50 vars in 791 bytes} [Thu Jul 14 11:58:26 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:35.160335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b186fc6-c8c8-430f-9886-600152775223 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:35.160864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b186fc6-c8c8-430f-9886-600152775223 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:35.160864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b186fc6-c8c8-430f-9886-600152775223 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:35.165866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1/2] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:35 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:58:35.171088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:35.171534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:35.171628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:35.239904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: INFO dbcounter [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] Registered counter for database keystone Jul 14 11:58:35.260896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG oslo_db.sqlalchemy.engines [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80487) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 14 11:58:35.267073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writer thread running {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 14 11:58:35.445056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG passlib.handlers.bcrypt [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80487) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 14 11:58:35.445478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG passlib.handlers.bcrypt [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80487) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 14 11:58:35.499351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:35.500941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:35.622736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 2/3] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:35 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 453 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 14 11:58:35.818692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b186fc6-c8c8-430f-9886-600152775223 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:35.889827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: INFO dbcounter [None req-1b186fc6-c8c8-430f-9886-600152775223 None None] Registered counter for database keystone Jul 14 11:58:35.909332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG oslo_db.sqlalchemy.engines [None req-1b186fc6-c8c8-430f-9886-600152775223 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80488) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 14 11:58:35.915617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writer thread running {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 14 11:58:36.036892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b186fc6-c8c8-430f-9886-600152775223 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:36.041227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00f6cdb-db39-479d-b4f7-2ddf7cabf118 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:36.041887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00f6cdb-db39-479d-b4f7-2ddf7cabf118 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:36.042178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00f6cdb-db39-479d-b4f7-2ddf7cabf118 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:36.042461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00f6cdb-db39-479d-b4f7-2ddf7cabf118 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:36.050546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00f6cdb-db39-479d-b4f7-2ddf7cabf118 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:36.102542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING keystone.common.rbac_enforcer.enforcer [None req-d00f6cdb-db39-479d-b4f7-2ddf7cabf118 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 14 11:58:36.102906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d00f6cdb-db39-479d-b4f7-2ddf7cabf118 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:36.112406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 2/4] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Jul 14 11:58:35 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 14 11:58:36.118458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:36.133381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bc2a74-9872-4db3-a048-ee36dc0893d8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:36.137738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10610477-e425-4d85-98f6-119e6348b906 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:36.138899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10610477-e425-4d85-98f6-119e6348b906 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:36.139184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10610477-e425-4d85-98f6-119e6348b906 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:36.139459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10610477-e425-4d85-98f6-119e6348b906 admin admin] PATH_INFO: `/v3/projects/1074c03ef1614e229d46654abff5220e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:36.143494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10610477-e425-4d85-98f6-119e6348b906 admin admin] RBAC: Authorizing `identity:get_project(project_id=1074c03ef1614e229d46654abff5220e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:36.194530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING keystone.common.rbac_enforcer.enforcer [None req-10610477-e425-4d85-98f6-119e6348b906 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 14 11:58:36.195124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10610477-e425-4d85-98f6-119e6348b906 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:36.199532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 3/5] 10.222.0.27 () {56 vars in 1207 bytes} [Thu Jul 14 11:58:36 2022] GET /identity/v3/projects/1074c03ef1614e229d46654abff5220e => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:38.203254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3ef78-c9b6-4938-9656-d1b49135d88f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:38.203765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3ef78-c9b6-4938-9656-d1b49135d88f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:38.203863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8e3ef78-c9b6-4938-9656-d1b49135d88f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:38.207923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 3/6] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:38 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:38.214024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b563a-cff2-41b3-88b3-38d686c1dfa2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:38.214419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b563a-cff2-41b3-88b3-38d686c1dfa2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:38.214587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340b563a-cff2-41b3-88b3-38d686c1dfa2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:38.233713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-340b563a-cff2-41b3-88b3-38d686c1dfa2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:38.235267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-340b563a-cff2-41b3-88b3-38d686c1dfa2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:38.243195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 4/7] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:38 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 14 11:58:38.265436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3ef78-c9b6-4938-9656-d1b49135d88f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:38.278160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8e3ef78-c9b6-4938-9656-d1b49135d88f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:38.282305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77f010b-b000-4d78-a6e5-c90b5cc283cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:38.283008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f010b-b000-4d78-a6e5-c90b5cc283cf admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:38.283295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f010b-b000-4d78-a6e5-c90b5cc283cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:38.283572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77f010b-b000-4d78-a6e5-c90b5cc283cf admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:38.292254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77f010b-b000-4d78-a6e5-c90b5cc283cf admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:38.294033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f77f010b-b000-4d78-a6e5-c90b5cc283cf admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:38.302956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 4/8] 10.222.0.27 () {58 vars in 1324 bytes} [Thu Jul 14 11:58:38 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 11:58:38.308464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340b563a-cff2-41b3-88b3-38d686c1dfa2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:38.317454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-340b563a-cff2-41b3-88b3-38d686c1dfa2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:38.322417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5102dbff-4775-4c04-95da-6416aa654b79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:38.323326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5102dbff-4775-4c04-95da-6416aa654b79 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:38.323615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5102dbff-4775-4c04-95da-6416aa654b79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:38.323892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5102dbff-4775-4c04-95da-6416aa654b79 admin admin] PATH_INFO: `/v3/users/984129f5bec74f43b4a666f392319525` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:38.325292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5102dbff-4775-4c04-95da-6416aa654b79 admin admin] RBAC: Authorizing `identity:get_user(user_id=984129f5bec74f43b4a666f392319525)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:38.327179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5102dbff-4775-4c04-95da-6416aa654b79 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:38.328752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 5/9] 10.222.0.27 () {56 vars in 1198 bytes} [Thu Jul 14 11:58:38 2022] GET /identity/v3/users/984129f5bec74f43b4a666f392319525 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:40.414669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f64245c-aa8d-4bf5-b4c6-c37926845ac2 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:40.415227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f64245c-aa8d-4bf5-b4c6-c37926845ac2 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:40.415329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f64245c-aa8d-4bf5-b4c6-c37926845ac2 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:40.416805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 5/10] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:58:40.421789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a3f85f-3f1f-4331-b156-68899c5ad2ee None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:40.422176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a3f85f-3f1f-4331-b156-68899c5ad2ee None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:40.422352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a3f85f-3f1f-4331-b156-68899c5ad2ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:40.435170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe88f282-c5a5-41cb-8085-68ca5968417f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:40.435456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe88f282-c5a5-41cb-8085-68ca5968417f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:40.435738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe88f282-c5a5-41cb-8085-68ca5968417f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:40.437291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 6/11] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:58:40.439091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-97a3f85f-3f1f-4331-b156-68899c5ad2ee None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:40.440270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-97a3f85f-3f1f-4331-b156-68899c5ad2ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:40.442101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:40.442453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:40.442736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:40.486748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG passlib.handlers.bcrypt [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80488) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 14 11:58:40.487201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG passlib.handlers.bcrypt [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80488) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 14 11:58:40.520999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 6/12] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:40.532929 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:40.534522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:40.545134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a3f85f-3f1f-4331-b156-68899c5ad2ee None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:40.569692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:40.635301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 7/13] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:58:40.651028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a3f85f-3f1f-4331-b156-68899c5ad2ee None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:40.658043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:40.686512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:40.687264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:40.687559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:40.687839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:40.699330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:40.710680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:40.711631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:58:40.712449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 7/14] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:58:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:58:40.719113 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:40.766127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e7f8e8-a861-4f24-916c-01cc902af434 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:40.768439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:40.801729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:40.804727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:40.805633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:40.805920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:40.806200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:40.820240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c128e1e1-9855-4915-8e35-2917c0ef16e7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:40.853991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:40.854659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:40.854987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:40.855268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:40.856021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:40.856331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:40.857733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:40.870051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 8/15] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:58:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:58:40.873514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:40.875175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:40.875504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:40.972771 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 8/16] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:58:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 14 11:58:40.973488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50076460-afa8-467d-851f-7ee37ecb3480 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:40.978569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:40.981296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:41.007807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:41.008685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:41.008973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:41.009246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] PATH_INFO: `/v3/users/984129f5bec74f43b4a666f392319525` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:41.010457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] RBAC: Authorizing `identity:get_user(user_id=984129f5bec74f43b4a666f392319525)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:41.011967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:41.013599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 9/17] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:58:40 2022] GET /identity/v3/users/984129f5bec74f43b4a666f392319525 => generated 247 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:58:41.018967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:41.034598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:41.077855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e586f3-1d72-48fa-aded-e7bb3bbb5389 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:41.111820 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:41.112610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:41.112902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:41.113179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:41.118628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75606591-578e-4d90-a852-acee14088eb6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:41.131655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:41.133144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:41.152550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:41.153343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:41.153628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:41.153903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:41.155131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:58:41.156455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:41.157381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:41.158376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:58:41.158376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:58:41.158376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:58:41.158376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:58:41.158653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:41.158730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 9/18] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:58:41.158836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 10/19] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:41.164719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:41.165164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:41.221898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:41.245866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:41.264349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4716d5-4eea-4d04-a971-15763bdc5173 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:41.264757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef673956-bbd5-4642-9a67-2be6b73c74ae None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:41.298558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e558fdf8-c800-4b88-a045-a63642d7d4a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:41.299251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e558fdf8-c800-4b88-a045-a63642d7d4a7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:41.299539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e558fdf8-c800-4b88-a045-a63642d7d4a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:41.299819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e558fdf8-c800-4b88-a045-a63642d7d4a7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:41.299903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b6e27c-59c8-4c3b-b64c-69c98c3e3a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:41.300518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b6e27c-59c8-4c3b-b64c-69c98c3e3a13 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:41.300656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e558fdf8-c800-4b88-a045-a63642d7d4a7 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.domain.id=default, user.id=984129f5bec74f43b4a666f392319525) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:41.300801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b6e27c-59c8-4c3b-b64c-69c98c3e3a13 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:41.300971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e558fdf8-c800-4b88-a045-a63642d7d4a7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:41.301062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b6e27c-59c8-4c3b-b64c-69c98c3e3a13 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:41.301788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b6e27c-59c8-4c3b-b64c-69c98c3e3a13 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:41.302097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b6e27c-59c8-4c3b-b64c-69c98c3e3a13 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:41.302462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e558fdf8-c800-4b88-a045-a63642d7d4a7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:41.303690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b6e27c-59c8-4c3b-b64c-69c98c3e3a13 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:41.310874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 10/20] 10.222.0.27 () {56 vars in 1322 bytes} [Thu Jul 14 11:58:41 2022] GET /identity/v3/role_assignments?user.id=984129f5bec74f43b4a666f392319525&scope.domain.id=default&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 233 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:58:41.325321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 11/21] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:58:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:58:43.763836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fefc0-ada1-4e3b-a6a7-54a6aa519ee9 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:43.764374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fefc0-ada1-4e3b-a6a7-54a6aa519ee9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:43.764374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-271fefc0-ada1-4e3b-a6a7-54a6aa519ee9 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:43.765905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 11/22] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:43.771189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd555a5-8c21-447c-99e9-8741bbbbc213 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:43.771628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd555a5-8c21-447c-99e9-8741bbbbc213 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:43.771960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd555a5-8c21-447c-99e9-8741bbbbc213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:43.792808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ccd555a5-8c21-447c-99e9-8741bbbbc213 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:43.793995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ccd555a5-8c21-447c-99e9-8741bbbbc213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:43.845513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926ac6a1-759c-49a7-9b6a-1d9d4a4d91e1 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:43.845829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926ac6a1-759c-49a7-9b6a-1d9d4a4d91e1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:43.846074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-926ac6a1-759c-49a7-9b6a-1d9d4a4d91e1 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:43.847475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 12/23] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:58:43.852336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df24f36d-604c-4b24-a14f-0546a928550a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:43.852625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df24f36d-604c-4b24-a14f-0546a928550a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:43.852908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df24f36d-604c-4b24-a14f-0546a928550a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:43.874859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-df24f36d-604c-4b24-a14f-0546a928550a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:43.876082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-df24f36d-604c-4b24-a14f-0546a928550a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:43.928521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 12/24] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:43.969297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd555a5-8c21-447c-99e9-8741bbbbc213 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:44.027015 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 13/25] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Jul 14 11:58:44.072387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df24f36d-604c-4b24-a14f-0546a928550a None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:44.115136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:44.157210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:44.163035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd555a5-8c21-447c-99e9-8741bbbbc213 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:44.224678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbd3980-4da8-4c3e-bc50-0227b88ceaf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:44.226361 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd3980-4da8-4c3e-bc50-0227b88ceaf8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.229766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd3980-4da8-4c3e-bc50-0227b88ceaf8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.230258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd3980-4da8-4c3e-bc50-0227b88ceaf8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.251415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbd3980-4da8-4c3e-bc50-0227b88ceaf8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:44.253491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df24f36d-604c-4b24-a14f-0546a928550a None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-7fbd3980-4da8-4c3e-bc50-0227b88ceaf8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:44.261394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:44.262230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:44.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:58:44.273206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 13/26] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:58:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 304 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:58:44.277369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffad3b-5909-4148-9954-68caab68d9d8 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.279558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffad3b-5909-4148-9954-68caab68d9d8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.280434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ffad3b-5909-4148-9954-68caab68d9d8 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.282681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 14/27] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:44 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:44.285163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffad3b-5909-4148-9954-68caab68d9d8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:44.323825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d32348b-ea35-4324-be3a-85484e8b5f59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:44.324779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32348b-ea35-4324-be3a-85484e8b5f59 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.325145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32348b-ea35-4324-be3a-85484e8b5f59 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.325504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d32348b-ea35-4324-be3a-85484e8b5f59 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.343645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d32348b-ea35-4324-be3a-85484e8b5f59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:44.345274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d32348b-ea35-4324-be3a-85484e8b5f59 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-6d32348b-ea35-4324-be3a-85484e8b5f59 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:58:44.383485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:44.384367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:44.385492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:58:44.385492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:58:44.385492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:58:44.385492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:58:44.385492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 14/28] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:58:44.392080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac8fe8-d686-424e-b562-445a4ad0b1ec None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.392450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac8fe8-d686-424e-b562-445a4ad0b1ec None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.392806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83ac8fe8-d686-424e-b562-445a4ad0b1ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.420657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ffad3b-5909-4148-9954-68caab68d9d8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:44.429567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-83ac8fe8-d686-424e-b562-445a4ad0b1ec None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:44.430822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-83ac8fe8-d686-424e-b562-445a4ad0b1ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:44.459993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:44.461076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.461559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.462036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.463031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:44.463570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:44.465369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:44.474682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 15/29] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:58:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:44.475524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:44.479785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:44.548089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 15/30] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:44.549551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ac8fe8-d686-424e-b562-445a4ad0b1ec None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:44.589760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:44.619945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa17e7a-7b5e-4fb3-b388-4838fc96c37c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:44.667273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552625d7-c49d-4c68-93d0-26507a55694f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:44.668416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552625d7-c49d-4c68-93d0-26507a55694f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.668904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552625d7-c49d-4c68-93d0-26507a55694f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.669397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552625d7-c49d-4c68-93d0-26507a55694f None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.670530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552625d7-c49d-4c68-93d0-26507a55694f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:44.671166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552625d7-c49d-4c68-93d0-26507a55694f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:44.672750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552625d7-c49d-4c68-93d0-26507a55694f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:44.677014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83ac8fe8-d686-424e-b562-445a4ad0b1ec None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:44.686944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 16/31] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:58:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:44.688939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b40a66-1bee-4fa7-b96d-06001c9d244b None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.689428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b40a66-1bee-4fa7-b96d-06001c9d244b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.689918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b40a66-1bee-4fa7-b96d-06001c9d244b None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.691942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 17/32] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:44.692970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b40a66-1bee-4fa7-b96d-06001c9d244b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:44.718519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:44.719441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.719726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.720001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] PATH_INFO: `/v3/users/984129f5bec74f43b4a666f392319525` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.725309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] RBAC: Authorizing `identity:get_user(user_id=984129f5bec74f43b4a666f392319525)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:44.725309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:44.731082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 16/33] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:58:44 2022] GET /identity/v3/users/984129f5bec74f43b4a666f392319525 => generated 247 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:44.731082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:44.777829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:44.840196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b40a66-1bee-4fa7-b96d-06001c9d244b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:44.875260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:44.877782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4425acff-1667-4cf9-b2a8-d5d1703c2454 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:44.878720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425acff-1667-4cf9-b2a8-d5d1703c2454 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.879254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425acff-1667-4cf9-b2a8-d5d1703c2454 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.879584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73ecdb11-29fe-47b9-8db8-3a3d6323daec None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:44.879972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4425acff-1667-4cf9-b2a8-d5d1703c2454 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.882163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4425acff-1667-4cf9-b2a8-d5d1703c2454 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:44.884013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4425acff-1667-4cf9-b2a8-d5d1703c2454 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:44.917591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0240daa7-0880-490d-a846-99919721f7df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:44.918268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0240daa7-0880-490d-a846-99919721f7df None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:44.918554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0240daa7-0880-490d-a846-99919721f7df None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:44.918864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0240daa7-0880-490d-a846-99919721f7df None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:44.919581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0240daa7-0880-490d-a846-99919721f7df None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:44.921038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0240daa7-0880-490d-a846-99919721f7df None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:45.087488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 18/34] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Jul 14 11:58:44 2022] POST /identity/v3/roles => generated 219 bytes in 395 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:58:45.089549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de8500-db77-4377-97d2-e77f2112ba88 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.090077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de8500-db77-4377-97d2-e77f2112ba88 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.090613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20de8500-db77-4377-97d2-e77f2112ba88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.109728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-20de8500-db77-4377-97d2-e77f2112ba88 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:45.111627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-20de8500-db77-4377-97d2-e77f2112ba88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:45.152487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 17/35] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 11:58:44 2022] POST /identity/v3/projects => generated 301 bytes in 425 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:58:45.153324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0240daa7-0880-490d-a846-99919721f7df None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:45.210353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 19/36] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:45.212325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d30a09-e92f-42b8-a468-3c1f377f8d76 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.212836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d30a09-e92f-42b8-a468-3c1f377f8d76 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.213339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d30a09-e92f-42b8-a468-3c1f377f8d76 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.214987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 20/37] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:45.222092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a08ac9-b0c2-4a5d-9874-c72b0aeec63b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.222633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a08ac9-b0c2-4a5d-9874-c72b0aeec63b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.223903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a08ac9-b0c2-4a5d-9874-c72b0aeec63b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.265548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c8a08ac9-b0c2-4a5d-9874-c72b0aeec63b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:45.269933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c8a08ac9-b0c2-4a5d-9874-c72b0aeec63b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:45.311019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0240daa7-0880-490d-a846-99919721f7df None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:45.348901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:45.361139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:45.363719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.363719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.363719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.365627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:45.367680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:45.371904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 18/38] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:45.371904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:45.401625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:45.408767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 21/39] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:45.408767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a08ac9-b0c2-4a5d-9874-c72b0aeec63b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:45.508150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7779e0-65d4-4f45-af2d-08849999a90c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:45.544568 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:45.545258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.545545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.545825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.546697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:45.548185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:45.631986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a08ac9-b0c2-4a5d-9874-c72b0aeec63b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:45.643708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 19/40] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Jul 14 11:58:45 2022] POST /identity/v3/roles => generated 225 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:58:45.644618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:45.680285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7d04d8f-6e25-423e-968b-7281ca46e804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:45.681193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d04d8f-6e25-423e-968b-7281ca46e804 None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.681478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d04d8f-6e25-423e-968b-7281ca46e804 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.681758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7d04d8f-6e25-423e-968b-7281ca46e804 None admin] PATH_INFO: `/v3/domains/default/users/984129f5bec74f43b4a666f392319525/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.683925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d04d8f-6e25-423e-968b-7281ca46e804 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=984129f5bec74f43b4a666f392319525, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:45.685535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7d04d8f-6e25-423e-968b-7281ca46e804 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:45.747866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:45.757941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 22/41] 10.222.0.27 () {58 vars in 1361 bytes} [Thu Jul 14 11:58:45 2022] PUT /identity/v3/domains/default/users/984129f5bec74f43b4a666f392319525/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 350 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:58:45.759329 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db9be7-7b05-4001-864e-89789a28b773 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.759658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db9be7-7b05-4001-864e-89789a28b773 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.759941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0db9be7-7b05-4001-864e-89789a28b773 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.761411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 23/42] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:45.767529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0d466-ddc5-408b-82c5-0f9286dab895 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.767854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0d466-ddc5-408b-82c5-0f9286dab895 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.768095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac0d466-ddc5-408b-82c5-0f9286dab895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.796405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b45b9a-bdaa-4778-8c9f-6ae0fd575eb0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:45.799101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3ac0d466-ddc5-408b-82c5-0f9286dab895 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:45.799101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3ac0d466-ddc5-408b-82c5-0f9286dab895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:45.836477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:45.850950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:45.852037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:45.852520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:45.853074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:45.854050 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:45.855823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:45.889002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 24/43] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:58:45.913563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0d466-ddc5-408b-82c5-0f9286dab895 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:45.919400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 20/44] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Jul 14 11:58:45 2022] POST /identity/v3/roles => generated 223 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:58:46.039192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac0d466-ddc5-408b-82c5-0f9286dab895 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:46.076658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f90293fe-b6e5-433f-9237-3b3bd80481ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:46.077465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90293fe-b6e5-433f-9237-3b3bd80481ce None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:46.077760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90293fe-b6e5-433f-9237-3b3bd80481ce None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:46.078049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f90293fe-b6e5-433f-9237-3b3bd80481ce None admin] PATH_INFO: `/v3/domains/default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:46.079283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90293fe-b6e5-433f-9237-3b3bd80481ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:46.080767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f90293fe-b6e5-433f-9237-3b3bd80481ce None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:46.082579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 25/45] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:46.088521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:46.190836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c38ffa3-ea5b-4e1e-802c-60862afae797 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:46.225342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a806ddcb-b805-4aba-a66e-c68124a4e720 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:46.226010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806ddcb-b805-4aba-a66e-c68124a4e720 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:46.226299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806ddcb-b805-4aba-a66e-c68124a4e720 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:46.226581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a806ddcb-b805-4aba-a66e-c68124a4e720 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:46.227336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806ddcb-b805-4aba-a66e-c68124a4e720 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:46.228799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a806ddcb-b805-4aba-a66e-c68124a4e720 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:46.239105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:46.301108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 21/46] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 11:58:46 2022] POST /identity/v3/projects => generated 312 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:58:47.812433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13b612-20fc-47ee-a78a-b407e1cb66bc None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:47.812943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13b612-20fc-47ee-a78a-b407e1cb66bc None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:47.813104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13b612-20fc-47ee-a78a-b407e1cb66bc None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:47.814514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 26/47] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:47.819806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5936f70-2e17-4b6f-aa8b-80d96f6f08b7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:47.820231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5936f70-2e17-4b6f-aa8b-80d96f6f08b7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:47.820489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5936f70-2e17-4b6f-aa8b-80d96f6f08b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:47.837596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-a5936f70-2e17-4b6f-aa8b-80d96f6f08b7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:47.839147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a5936f70-2e17-4b6f-aa8b-80d96f6f08b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:47.904468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 22/48] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:47.927670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c13b612-20fc-47ee-a78a-b407e1cb66bc None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:47.957633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:48.029597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c13b612-20fc-47ee-a78a-b407e1cb66bc None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:48.063305 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:48.064096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.064382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.064658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.075655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:48.083267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:48.084121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:58:48.084881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 27/49] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:58:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:58:48.090975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5936f70-2e17-4b6f-aa8b-80d96f6f08b7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:48.187048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5936f70-2e17-4b6f-aa8b-80d96f6f08b7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:48.210692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:48.223633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83d84638-74a7-4e4f-bf4d-e101bfa585b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:48.224363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d84638-74a7-4e4f-bf4d-e101bfa585b7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.224671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d84638-74a7-4e4f-bf4d-e101bfa585b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.224957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83d84638-74a7-4e4f-bf4d-e101bfa585b7 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.225713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d84638-74a7-4e4f-bf4d-e101bfa585b7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:48.226029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d84638-74a7-4e4f-bf4d-e101bfa585b7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:48.227466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83d84638-74a7-4e4f-bf4d-e101bfa585b7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:48.240054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 23/50] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:58:48 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:48.245647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:48.326569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a65b9a-27d7-48bd-a2ca-67e71ae42379 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.326811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a65b9a-27d7-48bd-a2ca-67e71ae42379 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.327107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a65b9a-27d7-48bd-a2ca-67e71ae42379 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.328491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 24/51] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:58:48.333294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd2424-0fa7-47e3-86c5-0b533a833ca9 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.333582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd2424-0fa7-47e3-86c5-0b533a833ca9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.334498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd2424-0fa7-47e3-86c5-0b533a833ca9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.337743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:48.348547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d514f5-0795-4a4c-a0de-4a5e8618a8cf None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:48.351105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-69bd2424-0fa7-47e3-86c5-0b533a833ca9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:48.352301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-69bd2424-0fa7-47e3-86c5-0b533a833ca9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:48.383663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:48.384549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.384836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.385114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] PATH_INFO: `/v3/users/984129f5bec74f43b4a666f392319525` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.386307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] RBAC: Authorizing `identity:get_user(user_id=984129f5bec74f43b4a666f392319525)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:48.387860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:48.389516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 28/52] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:58:48 2022] GET /identity/v3/users/984129f5bec74f43b4a666f392319525 => generated 247 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:58:48.394951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:48.420526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 25/53] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:58:48.444568 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bd2424-0fa7-47e3-86c5-0b533a833ca9 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:48.497750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beb134fd-ff85-4c09-991f-af2787897f1e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:48.534949 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:48.535771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.536058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.536335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] PATH_INFO: `/v3/domains/default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.537658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:48.539230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:48.540978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 29/54] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:58:48.547128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:48.576809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:48.647705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bd2424-0fa7-47e3-86c5-0b533a833ca9 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:48.651990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d6d6a3e-3b28-4bea-ad05-558d279da3bf None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:48.661839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:48.683275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:48.684074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.684363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.684645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.685805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:48.686475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb16cac-9da2-4134-b7d5-e4b6a3bcb8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:48.687162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb16cac-9da2-4134-b7d5-e4b6a3bcb8bf None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.687406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:48.687522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb16cac-9da2-4134-b7d5-e4b6a3bcb8bf None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.687725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb16cac-9da2-4134-b7d5-e4b6a3bcb8bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.688564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb16cac-9da2-4134-b7d5-e4b6a3bcb8bf None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.domain.id=default, user.id=984129f5bec74f43b4a666f392319525) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:48.688882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb16cac-9da2-4134-b7d5-e4b6a3bcb8bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:48.689329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 26/55] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:58:48.690332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb16cac-9da2-4134-b7d5-e4b6a3bcb8bf None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:48.695121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:48.713892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 30/56] 10.222.0.27 () {56 vars in 1322 bytes} [Thu Jul 14 11:58:48 2022] GET /identity/v3/role_assignments?user.id=984129f5bec74f43b4a666f392319525&scope.domain.id=default&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 530 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:48.824062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61431d51-2d5e-4201-b4f9-53e215d0a238 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:48.860192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95146635-cecd-4d07-a0ee-d2f4609443f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:48.860796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95146635-cecd-4d07-a0ee-d2f4609443f9 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:48.861088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95146635-cecd-4d07-a0ee-d2f4609443f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:48.861376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95146635-cecd-4d07-a0ee-d2f4609443f9 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:48.862095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95146635-cecd-4d07-a0ee-d2f4609443f9 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:48.863576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95146635-cecd-4d07-a0ee-d2f4609443f9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:48.940766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 27/57] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Jul 14 11:58:48 2022] POST /identity/v3/projects => generated 298 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 11:58:50.967148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec24dfe-f0cc-44ac-a059-2c1e491fd250 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:50.967661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec24dfe-f0cc-44ac-a059-2c1e491fd250 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:50.967771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec24dfe-f0cc-44ac-a059-2c1e491fd250 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:50.969079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 31/58] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:50.974111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2e3a1-f7e6-4b52-8282-2dc7f3fce284 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:50.975155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2e3a1-f7e6-4b52-8282-2dc7f3fce284 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:50.975486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df2e3a1-f7e6-4b52-8282-2dc7f3fce284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:50.992058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-5df2e3a1-f7e6-4b52-8282-2dc7f3fce284 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:50.993209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5df2e3a1-f7e6-4b52-8282-2dc7f3fce284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:51.018970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:51.058647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 28/59] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:51.083303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec24dfe-f0cc-44ac-a059-2c1e491fd250 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:51.173233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:51.183520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec24dfe-f0cc-44ac-a059-2c1e491fd250 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:51.216924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfc11f2-3d8f-4183-b01a-4ed5bf701482 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:51.217703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfc11f2-3d8f-4183-b01a-4ed5bf701482 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:51.217994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfc11f2-3d8f-4183-b01a-4ed5bf701482 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:51.218272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfc11f2-3d8f-4183-b01a-4ed5bf701482 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:51.219466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfc11f2-3d8f-4183-b01a-4ed5bf701482 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:51.220908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfc11f2-3d8f-4183-b01a-4ed5bf701482 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:51.222521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 32/60] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:51 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:51.228036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2e3a1-f7e6-4b52-8282-2dc7f3fce284 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:51.325674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df2e3a1-f7e6-4b52-8282-2dc7f3fce284 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:51.358637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a1fa7f-b6f1-44c2-8700-598fc742d468 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:51.359326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a1fa7f-b6f1-44c2-8700-598fc742d468 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:51.359615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a1fa7f-b6f1-44c2-8700-598fc742d468 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:51.359896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a1fa7f-b6f1-44c2-8700-598fc742d468 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:51.360608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a1fa7f-b6f1-44c2-8700-598fc742d468 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:51.362055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a1fa7f-b6f1-44c2-8700-598fc742d468 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:51.441895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 29/61] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:58:51 2022] POST /identity/v3/users => generated 275 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:58:54.672495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaabf9-0351-47bc-9049-b20244eaa822 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:54.673043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaabf9-0351-47bc-9049-b20244eaa822 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:54.673043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeaabf9-0351-47bc-9049-b20244eaa822 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:54.674416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 33/62] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:54.688347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83198820-26f3-4bbd-9dd8-e89413da5b92 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:54.689175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83198820-26f3-4bbd-9dd8-e89413da5b92 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:54.689623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83198820-26f3-4bbd-9dd8-e89413da5b92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:54.700854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac68f42-afdb-4eab-90a0-44ff747c4236 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:54.701547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac68f42-afdb-4eab-90a0-44ff747c4236 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:54.701966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ac68f42-afdb-4eab-90a0-44ff747c4236 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:54.707615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 34/63] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:54 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:54.716002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-83198820-26f3-4bbd-9dd8-e89413da5b92 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:54.717400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-83198820-26f3-4bbd-9dd8-e89413da5b92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:54.724722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3763b4-d58b-42be-8faa-01da25cd7d92 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:54.725195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3763b4-d58b-42be-8faa-01da25cd7d92 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:54.725616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3763b4-d58b-42be-8faa-01da25cd7d92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:54.750074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8e3763b4-d58b-42be-8faa-01da25cd7d92 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:54.751286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8e3763b4-d58b-42be-8faa-01da25cd7d92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:54.758149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:54.870866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 35/64] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:58:54.876335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 30/65] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:54.904266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3763b4-d58b-42be-8faa-01da25cd7d92 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:54.925530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efca6f6-cc28-4686-9ae1-937d610036de None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:54.926115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efca6f6-cc28-4686-9ae1-937d610036de None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:54.926588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efca6f6-cc28-4686-9ae1-937d610036de None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:54.929671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 31/66] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:54 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:58:54.935846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78f2dc0-5bfe-43b9-96f8-e5d982d13b44 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:54.936364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78f2dc0-5bfe-43b9-96f8-e5d982d13b44 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:54.936834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78f2dc0-5bfe-43b9-96f8-e5d982d13b44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:54.956604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e78f2dc0-5bfe-43b9-96f8-e5d982d13b44 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:54.958127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e78f2dc0-5bfe-43b9-96f8-e5d982d13b44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:55.005954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.018372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3763b4-d58b-42be-8faa-01da25cd7d92 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.051981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 32/67] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:55.053729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78f2dc0-5bfe-43b9-96f8-e5d982d13b44 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.059820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-713be38e-4f34-4fa8-bb5b-cd673b2d99c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.060702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713be38e-4f34-4fa8-bb5b-cd673b2d99c3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.061061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713be38e-4f34-4fa8-bb5b-cd673b2d99c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.061412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-713be38e-4f34-4fa8-bb5b-cd673b2d99c3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.073480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-713be38e-4f34-4fa8-bb5b-cd673b2d99c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-713be38e-4f34-4fa8-bb5b-cd673b2d99c3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:55.081825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:55.082687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:58:55.084119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 36/68] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:58:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:58:55.085060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea8a68-0e79-4e65-a081-e9f4a394f2b4 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.085135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea8a68-0e79-4e65-a081-e9f4a394f2b4 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.085729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ea8a68-0e79-4e65-a081-e9f4a394f2b4 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.086889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 37/69] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:55.089248 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ea8a68-0e79-4e65-a081-e9f4a394f2b4 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.152045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.170471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78f2dc0-5bfe-43b9-96f8-e5d982d13b44 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.195006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ea8a68-0e79-4e65-a081-e9f4a394f2b4 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.204043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.204850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.205139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.205416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.216504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:55.226475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:55.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:58:55.228196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:58:55.228781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe32ba2-f6a3-4477-9056-be27f907bded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.228878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 33/70] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:58:55.228944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe32ba2-f6a3-4477-9056-be27f907bded None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.228944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe32ba2-f6a3-4477-9056-be27f907bded None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.229056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.229118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe32ba2-f6a3-4477-9056-be27f907bded None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.243284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe32ba2-f6a3-4477-9056-be27f907bded None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-cfe32ba2-f6a3-4477-9056-be27f907bded None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:55.251411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:55.252392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:58:55.253153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 38/71] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:58:55.253781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4f23d1-4f75-46f9-a210-21a2f1b0292f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.254088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4f23d1-4f75-46f9-a210-21a2f1b0292f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.254375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4f23d1-4f75-46f9-a210-21a2f1b0292f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.270633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4c4f23d1-4f75-46f9-a210-21a2f1b0292f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:55.272476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4c4f23d1-4f75-46f9-a210-21a2f1b0292f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:55.274672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.332791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ac7703-bf3b-42bf-ba16-b53c261f1cc0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.338568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 39/72] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:55.339895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc189b-b90c-4a08-81c5-e61cc723195c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.340187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc189b-b90c-4a08-81c5-e61cc723195c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.340469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6bc189b-b90c-4a08-81c5-e61cc723195c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.341745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 40/73] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:55.342436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc189b-b90c-4a08-81c5-e61cc723195c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.377407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.378089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.378382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.378664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.379445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:55.379764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:55.381227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:55.389478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 34/74] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:55.390116 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.392616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.449630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6bc189b-b90c-4a08-81c5-e61cc723195c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.482958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e0d4d9a-dd2c-43b1-a0bd-3f017e55e8cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.483633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0d4d9a-dd2c-43b1-a0bd-3f017e55e8cc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.483929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0d4d9a-dd2c-43b1-a0bd-3f017e55e8cc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.484208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e0d4d9a-dd2c-43b1-a0bd-3f017e55e8cc None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.484962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0d4d9a-dd2c-43b1-a0bd-3f017e55e8cc None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:55.485272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0d4d9a-dd2c-43b1-a0bd-3f017e55e8cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:55.486680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e0d4d9a-dd2c-43b1-a0bd-3f017e55e8cc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:55.490621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880e04d2-fbde-42fc-a17f-656ecde1cfa9 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.492399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.496241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 41/75] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:55.497438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc4e73-2f9a-4f94-a04d-11062b6d9b4f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.497730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc4e73-2f9a-4f94-a04d-11062b6d9b4f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.498017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6dc4e73-2f9a-4f94-a04d-11062b6d9b4f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.514244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e6dc4e73-2f9a-4f94-a04d-11062b6d9b4f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:55.515470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e6dc4e73-2f9a-4f94-a04d-11062b6d9b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:55.525277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.525936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.526217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.526547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.527324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:55.527628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:55.529037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:55.536842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 35/76] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:55.537624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.586052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 42/77] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:55.587013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dc4e73-2f9a-4f94-a04d-11062b6d9b4f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.603791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.644550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dece3a3f-5b1e-4b3a-8421-464f0afb6fb8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.681119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.681914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.682203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.682486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.685528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6dc4e73-2f9a-4f94-a04d-11062b6d9b4f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.692443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:55.700626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:55.701582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:58:55.702344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 36/78] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:58:55.703003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.717342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.723126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.724047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.724832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.725098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] PATH_INFO: `/v3/users/984129f5bec74f43b4a666f392319525` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.726459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] RBAC: Authorizing `identity:get_user(user_id=984129f5bec74f43b4a666f392319525)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:55.728275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:55.729592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 43/79] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/users/984129f5bec74f43b4a666f392319525 => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:55.730352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.803984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368f397d-bd33-43fe-9722-f94083a74437 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.826339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.828987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d7f98a0-35c8-4358-a85d-09bcfeb01c88 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.837173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.838055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.840335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.840335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.854903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:55.856386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:55.858074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 37/80] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:55.858901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.862736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.863566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.863847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.864123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.865904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:55.867404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:55.869032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 44/81] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:55.869785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:55.933546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:55.962853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6798d20b-e1df-4e17-8fad-0dea8895f407 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.966951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e30ab1-c2dd-43a4-aa07-1b7cb9dbddd3 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:55.996891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:55.997691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:55.997976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:55.998258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:55.999468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.000218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.000882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.000984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.001168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.001446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.002196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:56.002509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 38/82] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:56.002652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.003303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.003914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.014343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 45/83] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:58:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:56.015173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.058856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.111227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0cd1a8-4a4c-45fd-904e-2feacc49a460 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:56.113265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1992ed41-3754-40cf-bc51-374e2dca86d9 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:56.138161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.145008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.145323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.145666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.146489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.146590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.146805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.146951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.147233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.213816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.214228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.215249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.215780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.261918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.312832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 46/84] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:58:56.312995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 39/85] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:58:56.313589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.313688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.343757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.414085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e839f7-fcf2-40f1-8549-e77d1ffd6dd5 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:56.414264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b863be9a-8e51-4595-9f7c-1a6e37fbb03f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:56.446840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.447114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.447507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.447764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.447849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.448048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.448134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.448328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.448758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.450208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.504791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 40/86] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:58:56 2022] POST /identity/v3/users => generated 289 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:58:56.505963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.514537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.525093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.526623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.558867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.620121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b920420-6545-4d3b-aeda-35d72329e6eb None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:56.653829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 47/87] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:58:56.654697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.660879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.662296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.662909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.663180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.664525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.666016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.667894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 41/88] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:56.668511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.777288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.789668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46bdf7a6-2770-4761-ae68-f1323122569c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:56.818379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b86ff37-fa50-448e-9a58-f321bab03f9e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:56.836953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:56.850559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.852062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.852572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.853049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.855050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.857775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.862078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:56.863070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:56.863361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:56.863639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:56.865393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:56.865727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 48/89] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:56.866822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.866925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:56.868872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 42/90] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:56.869645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:56.990347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2145d10f-9cdc-47cf-975f-9e13f4e64d6d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:57.008550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aacee49-ce10-4deb-9865-9a01a9f008d0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:57.027849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:57.028648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:57.028934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:57.029211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:57.030543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:57.031913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:57.033930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 49/91] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:57.034898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:57.059681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:57.061115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:57.061621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:57.062320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:57.080066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:57.103324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:57.145769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a979f2d-7353-47cb-9403-ce0d5a157eb6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:57.154181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:57.156724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:57.185563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:57.186232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:57.186518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:57.186834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:57.187690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=56dd0fcc7125401f8bc4841e48ca9167) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:57.188007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:57.189597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:57.197870 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 50/92] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:58:57 2022] GET /identity/v3/role_assignments?user.id=56dd0fcc7125401f8bc4841e48ca9167&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:57.198690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:57.291740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 43/93] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:58:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:58:57.292870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:57.367810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:57.406448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4192f544-eea3-4715-9533-64ca966b1d89 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:57.441479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:57.445521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8757f95-9a9e-4fcb-9fca-25d6eeec7ea7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:57.449783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:57.450523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:57.450834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:57.451150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:57.452006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=984129f5bec74f43b4a666f392319525) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:57.452323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:57.453841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:57.460339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 51/94] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:58:57 2022] GET /identity/v3/role_assignments?user.id=984129f5bec74f43b4a666f392319525&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:57.461333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:57.575682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:57.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:57.576593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:57.576877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:57.577740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] RBAC: Adding query filter params (role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=56dd0fcc7125401f8bc4841e48ca9167) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:57.578060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:57.579589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:57.594224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 44/95] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:58:57 2022] GET /identity/v3/role_assignments?user.id=56dd0fcc7125401f8bc4841e48ca9167&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=1ff35caa6d284df39cd287349f66c567 => generated 259 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:57.653079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90bf582-c388-4d4f-937a-355f39a49a6a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:57.720573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ade69b-10b3-4a01-a380-6fe89563b098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:57.721459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ade69b-10b3-4a01-a380-6fe89563b098 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:57.721744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ade69b-10b3-4a01-a380-6fe89563b098 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:57.722023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ade69b-10b3-4a01-a380-6fe89563b098 None admin] PATH_INFO: `/v3/projects/58077090af8b45c8a70613a6bc5245f8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:57.725838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ade69b-10b3-4a01-a380-6fe89563b098 None admin] RBAC: Authorizing `identity:get_project(project_id=58077090af8b45c8a70613a6bc5245f8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:57.728772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ade69b-10b3-4a01-a380-6fe89563b098 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:57.729166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 52/96] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:58:57 2022] GET /identity/v3/projects/58077090af8b45c8a70613a6bc5245f8 => generated 312 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:57.735816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:57.873753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40d1f67-bc86-43a4-9b3d-c5fe2277ad61 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:57.885715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:57.919119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a664a6-a5f5-4a44-968d-3ce20671368b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:57.919796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a664a6-a5f5-4a44-968d-3ce20671368b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:57.920085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a664a6-a5f5-4a44-968d-3ce20671368b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:57.920364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a664a6-a5f5-4a44-968d-3ce20671368b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:57.921267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a664a6-a5f5-4a44-968d-3ce20671368b None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=58077090af8b45c8a70613a6bc5245f8, user.id=56dd0fcc7125401f8bc4841e48ca9167) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:58:57.921587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a664a6-a5f5-4a44-968d-3ce20671368b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:57.923062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a664a6-a5f5-4a44-968d-3ce20671368b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:57.935274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 45/97] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:58:57 2022] GET /identity/v3/role_assignments?user.id=56dd0fcc7125401f8bc4841e48ca9167&scope.project.id=58077090af8b45c8a70613a6bc5245f8&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 259 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:59.275254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7573e9ea-e261-4a7a-a04b-e4856c5a7d0c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:59.275769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7573e9ea-e261-4a7a-a04b-e4856c5a7d0c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:59.275843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7573e9ea-e261-4a7a-a04b-e4856c5a7d0c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:59.277441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 53/98] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:58:59.285334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34da29f5-8328-4fd3-b47e-cdf053f9788b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:59.286173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34da29f5-8328-4fd3-b47e-cdf053f9788b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:59.287266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34da29f5-8328-4fd3-b47e-cdf053f9788b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:59.314745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-34da29f5-8328-4fd3-b47e-cdf053f9788b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:59.316244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-34da29f5-8328-4fd3-b47e-cdf053f9788b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:59.363108 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446c0ec-3678-4fe3-ab6c-50c21158cffb None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:59.363695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446c0ec-3678-4fe3-ab6c-50c21158cffb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:59.364150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9446c0ec-3678-4fe3-ab6c-50c21158cffb None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:59.366364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 54/99] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:58:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:58:59.371091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d5091-7397-4444-8557-758e6c131f35 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:59.371565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d5091-7397-4444-8557-758e6c131f35 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:59.372023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86d5091-7397-4444-8557-758e6c131f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:59.403504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f86d5091-7397-4444-8557-758e6c131f35 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:58:59.406136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f86d5091-7397-4444-8557-758e6c131f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:58:59.449024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 46/100] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:58:59.481258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:59.492602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 55/101] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:58:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:58:59.502049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34da29f5-8328-4fd3-b47e-cdf053f9788b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:59.532234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86d5091-7397-4444-8557-758e6c131f35 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:59.675220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:59.685661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34da29f5-8328-4fd3-b47e-cdf053f9788b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:59.693445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86d5091-7397-4444-8557-758e6c131f35 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:58:59.743519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:59.744675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:59.745153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:59.745618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:59.747080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:58:59.750825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:58:59.757657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 56/102] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:58:59 2022] GET /identity/v3/domains/default => generated 199 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:58:59.758755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:59.761710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:58:59.762537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:58:59.762844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:58:59.763126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:58:59.796217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:58:59.810271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:58:59.811291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:58:59.812062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 47/103] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:58:59 2022] GET /identity/v3/roles/reader => generated 84 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:58:59.819784 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:58:59.860386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:58:59.928014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facb8c2f-f48d-47ee-b7df-ae8dd27930b0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:00.023571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:00.024517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.024806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.025089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.025817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:00.027358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:00.042968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5e11fa-2867-4b06-a5ed-e09772e65837 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:00.082860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:00.082860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.082860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.082860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.083337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:00.083337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:00.084059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:00.093140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 48/104] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:58:59 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:00.098044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:00.135318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:00.162300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 57/105] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 11:58:59 2022] POST /identity/v3/projects => generated 302 bytes in 404 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:00.232333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d2dc51-1b22-4b4c-b032-7240933a5804 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:00.277897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d6f716-fee0-4697-afcc-5e3b64c335a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:00.279245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d6f716-fee0-4697-afcc-5e3b64c335a7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.279741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d6f716-fee0-4697-afcc-5e3b64c335a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.280241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d6f716-fee0-4697-afcc-5e3b64c335a7 None admin] PATH_INFO: `/v3/users/25d5254a28dc4049951e1b7dfd25ccd2` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.296176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d6f716-fee0-4697-afcc-5e3b64c335a7 None admin] RBAC: Authorizing `identity:get_user(user_id=25d5254a28dc4049951e1b7dfd25ccd2)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:00.298582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d6f716-fee0-4697-afcc-5e3b64c335a7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:00.300828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 49/106] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:00 2022] GET /identity/v3/users/25d5254a28dc4049951e1b7dfd25ccd2 => generated 289 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:00.307087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:00.437784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e18637-d483-4c07-b86a-674f3787da3a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:00.453657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:00.483739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c098d-b6a2-4215-befd-58dfd607d09d None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.484039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c098d-b6a2-4215-befd-58dfd607d09d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.484597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-308c098d-b6a2-4215-befd-58dfd607d09d None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.486192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 50/107] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:59:00.490122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:00.490122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.490122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.490122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.492716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b818f64-58bb-4fdb-b85e-8eb97553052d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.493248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b818f64-58bb-4fdb-b85e-8eb97553052d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.493790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b818f64-58bb-4fdb-b85e-8eb97553052d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.535907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1b818f64-58bb-4fdb-b85e-8eb97553052d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:00.537268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1b818f64-58bb-4fdb-b85e-8eb97553052d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:00.570431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:00.572313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:00.673099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 51/108] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:00.674727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032445cf-89b4-4433-b2f4-b8102f02f88b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.675339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032445cf-89b4-4433-b2f4-b8102f02f88b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.675819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032445cf-89b4-4433-b2f4-b8102f02f88b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.678482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 52/109] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:00.684334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fdffea-809c-4808-ae89-516db60135f5 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.684865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fdffea-809c-4808-ae89-516db60135f5 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.685604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08fdffea-809c-4808-ae89-516db60135f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.720329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-08fdffea-809c-4808-ae89-516db60135f5 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:00.721940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-08fdffea-809c-4808-ae89-516db60135f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:00.731893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:00.781518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 58/110] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:00 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:00.783164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:00.795803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:00.896842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eeb2ce-051a-4c11-b8c2-39e7a93bad7d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:00.917035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 53/111] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 234 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:00.918254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fdffea-809c-4808-ae89-516db60135f5 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:00.954369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67cc0f4-36bb-4aed-ad24-828f804303d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:00.955382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67cc0f4-36bb-4aed-ad24-828f804303d8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.955673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67cc0f4-36bb-4aed-ad24-828f804303d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.955959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67cc0f4-36bb-4aed-ad24-828f804303d8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.968139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67cc0f4-36bb-4aed-ad24-828f804303d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-f67cc0f4-36bb-4aed-ad24-828f804303d8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:00.977739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:00.978560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:00.979373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 59/112] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:00.984329 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2aae8-f6bb-4b9f-80c5-0c89cde93dd9 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:00.984955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2aae8-f6bb-4b9f-80c5-0c89cde93dd9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:00.985432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2aae8-f6bb-4b9f-80c5-0c89cde93dd9 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:00.987015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 60/113] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:00 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:00.991834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec2aae8-f6bb-4b9f-80c5-0c89cde93dd9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.038412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08fdffea-809c-4808-ae89-516db60135f5 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.071160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.073716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1eb0571-7c08-49cc-bde5-4e425907c4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.074578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb0571-7c08-49cc-bde5-4e425907c4b1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.074887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb0571-7c08-49cc-bde5-4e425907c4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.075163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1eb0571-7c08-49cc-bde5-4e425907c4b1 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.076339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb0571-7c08-49cc-bde5-4e425907c4b1 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:01.077922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1eb0571-7c08-49cc-bde5-4e425907c4b1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:01.085269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 54/114] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:00 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:01.085269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094cefa-57a8-4a52-bb90-c90b6518c00f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.085269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094cefa-57a8-4a52-bb90-c90b6518c00f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.085269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0094cefa-57a8-4a52-bb90-c90b6518c00f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.085269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 55/115] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:01 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:01.087097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094cefa-57a8-4a52-bb90-c90b6518c00f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.099450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.112140 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec2aae8-f6bb-4b9f-80c5-0c89cde93dd9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.151358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7764962-338a-41a0-925d-b99c7c03a436 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.152437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7764962-338a-41a0-925d-b99c7c03a436 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.152728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7764962-338a-41a0-925d-b99c7c03a436 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.153025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7764962-338a-41a0-925d-b99c7c03a436 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.169265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7764962-338a-41a0-925d-b99c7c03a436 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-f7764962-338a-41a0-925d-b99c7c03a436 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:01.180450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:01.181318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 61/116] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:00 2022] GET /identity/v3/roles/admin => generated 83 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:01.187260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c1954f-e02d-4e45-bb38-6a2cb2f03a3b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.188002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c1954f-e02d-4e45-bb38-6a2cb2f03a3b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.188002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c1954f-e02d-4e45-bb38-6a2cb2f03a3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.196667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0094cefa-57a8-4a52-bb90-c90b6518c00f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.205418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-80c1954f-e02d-4e45-bb38-6a2cb2f03a3b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:01.206845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-80c1954f-e02d-4e45-bb38-6a2cb2f03a3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:01.233608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.234200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.234495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.234799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.235604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:01.235848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:01.237219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:01.245565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 56/117] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:01.250733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.278853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 62/118] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:01.288555 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3838182e-6e71-4de5-b30d-b3c66361d0e1 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.289951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3838182e-6e71-4de5-b30d-b3c66361d0e1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.290440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3838182e-6e71-4de5-b30d-b3c66361d0e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.309901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3838182e-6e71-4de5-b30d-b3c66361d0e1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:01.311201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3838182e-6e71-4de5-b30d-b3c66361d0e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:01.327480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.359943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c435c29c-1140-40ea-8d5a-637ef65a3786 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.370132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.382518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 63/119] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:01.383394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3838182e-6e71-4de5-b30d-b3c66361d0e1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.406038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.406891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.407182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.407464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.408334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] RBAC: Adding query filter params (role.id=ae7f6ead5e04485fa45d5b011096569d, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=25d5254a28dc4049951e1b7dfd25ccd2) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:01.408661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:01.410108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:01.415991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 57/120] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/role_assignments?user.id=25d5254a28dc4049951e1b7dfd25ccd2&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=ae7f6ead5e04485fa45d5b011096569d => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:01.416591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.524853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3838182e-6e71-4de5-b30d-b3c66361d0e1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.543052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d90bc95-98bf-4e47-bbe5-d2999beb2d8a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.583104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.584053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.584158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.584394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.585431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:01.585431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:01.586965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:01.592457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.594428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.595687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.595687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.596343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:01.596933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 64/121] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:01.597554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.597854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:01.605651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 58/122] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:01.605651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.605651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.675715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.721085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bfa6a4-db75-4042-a5b1-6735a48a4731 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.743149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f1e3e3-e0b0-4c22-941a-7b5ead73b9e6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.774045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.775461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.775931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.776763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.781095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.782004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.782467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.782969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.790390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:01.794940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:01.799869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:01.800883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:01.801840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:01.804487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 59/123] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:01.805404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:01.806181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:01.807068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:01.807852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 65/124] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:01.808488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.885787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.931990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87bde54-86dc-4ab5-9b53-fca93700a544 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.942365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a56640-9508-47bc-8912-62457832d059 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:01.949004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:01.979344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.979895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:01.980360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.980654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.980938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] PATH_INFO: `/v3/users/984129f5bec74f43b4a666f392319525` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.981900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:01.982160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] RBAC: Authorizing `identity:get_user(user_id=984129f5bec74f43b4a666f392319525)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:01.982684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:01.983219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:01.983702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:01.984875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:01.990564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 60/125] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/users/984129f5bec74f43b4a666f392319525 => generated 247 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:01.990564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:01.991701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:01.993635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 66/126] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:01.994281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.122096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed2f0c7-a9ef-47e1-b9f6-4828f08fa219 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.122403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6bd291-4e75-49c1-b75d-15979283cef7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.159039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:02.159223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.160101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.160209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.160552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.161239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:02.161587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.163067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.167228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.176358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.176358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.176358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.176358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:02.176358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.176358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.177302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 67/127] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:02.177302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.181257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 61/128] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:02.182201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.219892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:02.280558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721afaf6-60d0-4837-af5f-ee4e1f276a45 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.289304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b0f7675-891c-48c1-8a38-f43c6f2407e3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.319400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.320904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.321337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.321759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.323180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.324890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.328647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.329085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.329085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.329255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/56dd0fcc7125401f8bc4841e48ca9167/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.329936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 68/129] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:02 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:02.330898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.336356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] RBAC: Authorizing `identity:create_grant(project_id=696f929ba4b64cd88ceb4cdaa2f61d36, user_id=56dd0fcc7125401f8bc4841e48ca9167, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.339183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.423320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 62/130] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:02 2022] PUT /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/56dd0fcc7125401f8bc4841e48ca9167/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:02.424485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.455578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2099398c-d0de-492b-9f09-e74d507d6acd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.481812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:02.502831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:02.546670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.548048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.548483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.548911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.550381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.552834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.560508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92478ac-b21f-48eb-b1d0-6e62b96e33c3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.560793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 69/131] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:02 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:02.561904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.606334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.607290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.607579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.607865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.609210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.610727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.612955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 63/132] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:02 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:02.613774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.713735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5df7b62-72e7-45eb-96d3-7c18d4725c2c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.738388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553e6a86-38e9-494d-82a9-b89753042821 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:02.755127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5b2e34-f346-461d-b64d-1f420e1eb703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.756660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5b2e34-f346-461d-b64d-1f420e1eb703 None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.757183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5b2e34-f346-461d-b64d-1f420e1eb703 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.757639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5b2e34-f346-461d-b64d-1f420e1eb703 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/984129f5bec74f43b4a666f392319525/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.760021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5b2e34-f346-461d-b64d-1f420e1eb703 None admin] RBAC: Authorizing `identity:create_grant(project_id=696f929ba4b64cd88ceb4cdaa2f61d36, user_id=984129f5bec74f43b4a666f392319525, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.761887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5b2e34-f346-461d-b64d-1f420e1eb703 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.763656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:02.779805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:02.780705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.780993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.781272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] PATH_INFO: `/v3/projects/58077090af8b45c8a70613a6bc5245f8` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.782455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] RBAC: Authorizing `identity:get_project(project_id=58077090af8b45c8a70613a6bc5245f8)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:02.784120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:02.786114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 64/133] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:02 2022] GET /identity/v3/projects/58077090af8b45c8a70613a6bc5245f8 => generated 312 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:02.789742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.850191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 70/134] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:02 2022] PUT /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/984129f5bec74f43b4a666f392319525/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:02.857617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f061c-0f45-4a4d-a2e3-5d448ce97ce1 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:02.858258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f061c-0f45-4a4d-a2e3-5d448ce97ce1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:02.858818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209f061c-0f45-4a4d-a2e3-5d448ce97ce1 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:02.862172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 71/135] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:02 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:02.863594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209f061c-0f45-4a4d-a2e3-5d448ce97ce1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:02.928930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:02.972503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d027c25-ee31-4f42-b62b-b8a36227581d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:03.029633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209f061c-0f45-4a4d-a2e3-5d448ce97ce1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:03.034860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2e4e0b2-ca82-4d02-b074-1b52db87856e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:03.036064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e4e0b2-ca82-4d02-b074-1b52db87856e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.037238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e4e0b2-ca82-4d02-b074-1b52db87856e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.037721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2e4e0b2-ca82-4d02-b074-1b52db87856e None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.039297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e4e0b2-ca82-4d02-b074-1b52db87856e None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:03.040985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2e4e0b2-ca82-4d02-b074-1b52db87856e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:03.043267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 65/136] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:02 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:03.044706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7406f22-16dd-4671-a833-c47fc46eccb4 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.045172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7406f22-16dd-4671-a833-c47fc46eccb4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.045624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7406f22-16dd-4671-a833-c47fc46eccb4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.064498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d7406f22-16dd-4671-a833-c47fc46eccb4 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:03.066043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d7406f22-16dd-4671-a833-c47fc46eccb4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:03.080543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:03.081842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.082408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.082884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] PATH_INFO: `/v3/projects/58077090af8b45c8a70613a6bc5245f8/users/56dd0fcc7125401f8bc4841e48ca9167/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.087001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] RBAC: Authorizing `identity:create_grant(project_id=58077090af8b45c8a70613a6bc5245f8, user_id=56dd0fcc7125401f8bc4841e48ca9167, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:03.089613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:03.126497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:03.142009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 66/137] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:03.142803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7406f22-16dd-4671-a833-c47fc46eccb4 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:03.169271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 72/138] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:02 2022] PUT /identity/v3/projects/58077090af8b45c8a70613a6bc5245f8/users/56dd0fcc7125401f8bc4841e48ca9167/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:03.193337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:03.271437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7406f22-16dd-4671-a833-c47fc46eccb4 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:03.358642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d040c42-f8c0-4b82-8047-901aaf2824b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:03.359601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d040c42-f8c0-4b82-8047-901aaf2824b1 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.359891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d040c42-f8c0-4b82-8047-901aaf2824b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.360175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d040c42-f8c0-4b82-8047-901aaf2824b1 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/56dd0fcc7125401f8bc4841e48ca9167/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.371783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d040c42-f8c0-4b82-8047-901aaf2824b1 None admin] RBAC: Authorizing `identity:create_grant(project_id=696f929ba4b64cd88ceb4cdaa2f61d36, user_id=56dd0fcc7125401f8bc4841e48ca9167, role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:03.373304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d040c42-f8c0-4b82-8047-901aaf2824b1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:03.407722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:03.413533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10190745-4a86-4c7f-b32e-44478f3f376e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:03.453346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:03.454508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.455012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.455482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.456882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:03.459301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:03.461495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 73/139] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:03 2022] GET /identity/v3/domains/default => generated 199 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:03.466755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:03.495544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 67/140] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:03 2022] PUT /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/56dd0fcc7125401f8bc4841e48ca9167/roles/1ff35caa6d284df39cd287349f66c567 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:03.603114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5813a2da-c6cf-445d-9b1e-05947ac4b872 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:03.635548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-883cf1ee-1e14-453a-b0b3-fae41331f9b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:03.636440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883cf1ee-1e14-453a-b0b3-fae41331f9b4 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.636724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883cf1ee-1e14-453a-b0b3-fae41331f9b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.637056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-883cf1ee-1e14-453a-b0b3-fae41331f9b4 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.637794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883cf1ee-1e14-453a-b0b3-fae41331f9b4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:03.639285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-883cf1ee-1e14-453a-b0b3-fae41331f9b4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:03.741707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 74/141] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:03 2022] POST /identity/v3/users => generated 283 bytes in 275 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 11:59:03.811760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4b038-3d10-4855-8b19-5eed079cff9f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.812103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4b038-3d10-4855-8b19-5eed079cff9f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.812328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d4b038-3d10-4855-8b19-5eed079cff9f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.813938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 68/142] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:03.821310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4fd6c-8a1c-432f-9c88-04a8d032fc41 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:03.821916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4fd6c-8a1c-432f-9c88-04a8d032fc41 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:03.822201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfb4fd6c-8a1c-432f-9c88-04a8d032fc41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:03.851963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bfb4fd6c-8a1c-432f-9c88-04a8d032fc41 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:03.853266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bfb4fd6c-8a1c-432f-9c88-04a8d032fc41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:03.877862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:03.949896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 75/143] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:03.972663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d4b038-3d10-4855-8b19-5eed079cff9f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:04.026191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:04.086842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d4b038-3d10-4855-8b19-5eed079cff9f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:04.131928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:04.132663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:04.132950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:04.133231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:04.143952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:04.153075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:04.154070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:04.156636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 69/144] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:03 2022] GET /identity/v3/roles/reader => generated 84 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:04.163025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4fd6c-8a1c-432f-9c88-04a8d032fc41 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:04.311896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfb4fd6c-8a1c-432f-9c88-04a8d032fc41 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:04.357847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:04.358914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:04.359425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:04.359915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:04.360866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] RBAC: Adding query filter params (name=reader) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:04.361366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:04.363083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:04.373143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 76/145] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:04 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:04.379983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:04.517176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ceb9bac-25a0-4ff6-98c0-47e02afda187 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:04.525104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:04.560832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:04.561724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:04.562014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:04.562359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] PATH_INFO: `/v3/users/25d5254a28dc4049951e1b7dfd25ccd2` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:04.563681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] RBAC: Authorizing `identity:get_user(user_id=25d5254a28dc4049951e1b7dfd25ccd2)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:04.565145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:04.566895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 70/146] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:04 2022] GET /identity/v3/users/25d5254a28dc4049951e1b7dfd25ccd2 => generated 289 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:04.572895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:04.598208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:04.704152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439a5905-254f-407d-ba8c-79601802f17a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:04.753203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d57ce2-e21d-4c91-830f-98f1b45535bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:04.754225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d57ce2-e21d-4c91-830f-98f1b45535bf None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:04.754716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d57ce2-e21d-4c91-830f-98f1b45535bf None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:04.755241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d57ce2-e21d-4c91-830f-98f1b45535bf None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:04.760220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d57ce2-e21d-4c91-830f-98f1b45535bf None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:04.761867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d57ce2-e21d-4c91-830f-98f1b45535bf None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:04.763676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 77/147] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:04 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:04.770203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:04.896410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d51f66f5-0470-44de-909d-1f7aa8941caa None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:04.942090 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71fc3102-a169-4e27-93f1-09ef679cb4c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:04.943425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc3102-a169-4e27-93f1-09ef679cb4c7 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:04.943881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc3102-a169-4e27-93f1-09ef679cb4c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:04.944046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71fc3102-a169-4e27-93f1-09ef679cb4c7 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/25d5254a28dc4049951e1b7dfd25ccd2/roles/ae7f6ead5e04485fa45d5b011096569d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:04.946104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fc3102-a169-4e27-93f1-09ef679cb4c7 None admin] RBAC: Authorizing `identity:create_grant(project_id=696f929ba4b64cd88ceb4cdaa2f61d36, user_id=25d5254a28dc4049951e1b7dfd25ccd2, role_id=ae7f6ead5e04485fa45d5b011096569d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:04.947724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71fc3102-a169-4e27-93f1-09ef679cb4c7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:05.027183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 71/148] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:04 2022] PUT /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/users/25d5254a28dc4049951e1b7dfd25ccd2/roles/ae7f6ead5e04485fa45d5b011096569d => generated 0 bytes in 257 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:05.234637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5363001-9f69-4c9f-a9ba-ae754de8974c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:05.235047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5363001-9f69-4c9f-a9ba-ae754de8974c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:05.235244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5363001-9f69-4c9f-a9ba-ae754de8974c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:05.236631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 78/149] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:05.242195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c17730-e823-48ad-856c-7d28388e273d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:05.242494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c17730-e823-48ad-856c-7d28388e273d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:05.242800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49c17730-e823-48ad-856c-7d28388e273d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:05.260594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-49c17730-e823-48ad-856c-7d28388e273d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:05.261813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-49c17730-e823-48ad-856c-7d28388e273d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:05.294991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:05.348348 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 72/150] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:05.386623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5363001-9f69-4c9f-a9ba-ae754de8974c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:05.490708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:05.511176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5363001-9f69-4c9f-a9ba-ae754de8974c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:05.553168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:05.553892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:05.554177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:05.554452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:05.572378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:05.580372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:05.582467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:05.583947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 79/151] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:05.589202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c17730-e823-48ad-856c-7d28388e273d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:05.736171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49c17730-e823-48ad-856c-7d28388e273d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:05.782316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60842006-2bee-4af9-9554-356c900e916a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:05.783017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60842006-2bee-4af9-9554-356c900e916a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:05.783304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60842006-2bee-4af9-9554-356c900e916a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:05.783583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60842006-2bee-4af9-9554-356c900e916a None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:05.784337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60842006-2bee-4af9-9554-356c900e916a None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:05.784647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60842006-2bee-4af9-9554-356c900e916a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:05.786082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60842006-2bee-4af9-9554-356c900e916a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:05.795941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 73/152] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:05.801435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:05.913799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf456ea7-e204-451c-a910-37e211aa9971 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:05.954637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2caa281b-2ebf-4c67-ae46-68ad9b73be83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:05.955551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa281b-2ebf-4c67-ae46-68ad9b73be83 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:05.955842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa281b-2ebf-4c67-ae46-68ad9b73be83 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:05.956118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2caa281b-2ebf-4c67-ae46-68ad9b73be83 None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:05.963574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa281b-2ebf-4c67-ae46-68ad9b73be83 None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:05.965280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2caa281b-2ebf-4c67-ae46-68ad9b73be83 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:05.967571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 80/153] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:05 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:05.972835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60842006-2bee-4af9-9554-356c900e916a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:06.020280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:06.099617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60842006-2bee-4af9-9554-356c900e916a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:06.140754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:06.141414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:06.141698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:06.141977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:06.241237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:06.242720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:06.276620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:06.340012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270320c1-f5a5-4bd5-9374-f33c3ba2ced2 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:06.340660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270320c1-f5a5-4bd5-9374-f33c3ba2ced2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:06.341142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270320c1-f5a5-4bd5-9374-f33c3ba2ced2 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:06.346398 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 81/154] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:06 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:06.347834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bf6238-3626-427b-b16a-46e5f823a48e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:06.348332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bf6238-3626-427b-b16a-46e5f823a48e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:06.349415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12bf6238-3626-427b-b16a-46e5f823a48e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:06.376897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 74/155] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:06.382406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-12bf6238-3626-427b-b16a-46e5f823a48e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:06.382663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:06.384058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-12bf6238-3626-427b-b16a-46e5f823a48e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:06.391777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:06.539907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 82/156] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:06.557565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724a3b5b-bb9f-4d14-80b9-59b7d080b49c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:06.575111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bf6238-3626-427b-b16a-46e5f823a48e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:06.602429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:06.616273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:06.617958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:06.617958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:06.617958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:06.623556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:06.625031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:06.626803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 75/157] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:06 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:06.633914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:06.768811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12bf6238-3626-427b-b16a-46e5f823a48e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:06.844119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:06.844959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:06.845247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:06.845522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:06.862418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bed7e1-0a0f-4f81-99f3-fc32f69df2d1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:06.864282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:06.871709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:06.879650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:06.887616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:06.888391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 83/158] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 307 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:06.893430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:06.927978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:06.929496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:06.930050 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:06.930474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:06.931660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=56dd0fcc7125401f8bc4841e48ca9167) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:06.932138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:06.934078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:06.949519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 76/159] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:06 2022] GET /identity/v3/role_assignments?user.id=56dd0fcc7125401f8bc4841e48ca9167&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 608 bytes in 316 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:07.070256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065b645-3ddb-4a8d-821f-740307cd8b87 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:07.131007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:07.131614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.131894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.132206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:07.132967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:07.133275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:07.134679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:07.147469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 84/160] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:07.153956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:07.236232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:07.283826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d79120-b3c9-4b0a-a345-7936f2b8c1f2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:07.320890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e3be8a-0a26-488d-a612-b03a5be870f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:07.321772 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e3be8a-0a26-488d-a612-b03a5be870f2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.322049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e3be8a-0a26-488d-a612-b03a5be870f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.322316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e3be8a-0a26-488d-a612-b03a5be870f2 None admin] PATH_INFO: `/v3/users/984129f5bec74f43b4a666f392319525` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:07.323550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e3be8a-0a26-488d-a612-b03a5be870f2 None admin] RBAC: Authorizing `identity:get_user(user_id=984129f5bec74f43b4a666f392319525)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:07.325001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e3be8a-0a26-488d-a612-b03a5be870f2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:07.326631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 77/161] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:07 2022] GET /identity/v3/users/984129f5bec74f43b4a666f392319525 => generated 247 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:07.332480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:07.415402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:07.437672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af983b8-adb6-4eb2-90ac-d743392c0fb2 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.437989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af983b8-adb6-4eb2-90ac-d743392c0fb2 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.438221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af983b8-adb6-4eb2-90ac-d743392c0fb2 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:07.439620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 78/162] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:59:07.447575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c330c011-3e78-40fc-ad09-9a73af5fd3d3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.447854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c330c011-3e78-40fc-ad09-9a73af5fd3d3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.448129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c330c011-3e78-40fc-ad09-9a73af5fd3d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:07.475697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c330c011-3e78-40fc-ad09-9a73af5fd3d3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:07.476889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c330c011-3e78-40fc-ad09-9a73af5fd3d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:07.492085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a6a2d-4b68-4778-871f-241887317cf3 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:07.576039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a62f2b7-c3e8-4048-b623-d978afb2db0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:07.576700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62f2b7-c3e8-4048-b623-d978afb2db0f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.576976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62f2b7-c3e8-4048-b623-d978afb2db0f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.577252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a62f2b7-c3e8-4048-b623-d978afb2db0f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:07.628897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 79/163] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:07.662070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c330c011-3e78-40fc-ad09-9a73af5fd3d3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:07.715878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a62f2b7-c3e8-4048-b623-d978afb2db0f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:07.717372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a62f2b7-c3e8-4048-b623-d978afb2db0f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:07.763298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:07.850137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:07.861892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c330c011-3e78-40fc-ad09-9a73af5fd3d3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:07.929524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 85/164] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 597 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:07.932104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7baf3e-a447-42cf-a703-ebd626c2c03f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.933426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7baf3e-a447-42cf-a703-ebd626c2c03f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.934311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7baf3e-a447-42cf-a703-ebd626c2c03f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:07.936728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce17571e-8c36-4212-b6a6-3584661b0549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:07.937723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17571e-8c36-4212-b6a6-3584661b0549 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.938183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17571e-8c36-4212-b6a6-3584661b0549 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.938653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce17571e-8c36-4212-b6a6-3584661b0549 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:07.940414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 86/165] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:07 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:07.942512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7baf3e-a447-42cf-a703-ebd626c2c03f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:07.963701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce17571e-8c36-4212-b6a6-3584661b0549 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-ce17571e-8c36-4212-b6a6-3584661b0549 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:07.975973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:07.976861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:07.977761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 80/166] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 317 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:07.978467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f96ca-67e7-44ad-bab8-778e29cbacfb None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:07.979194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f96ca-67e7-44ad-bab8-778e29cbacfb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:07.979704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-138f96ca-67e7-44ad-bab8-778e29cbacfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.022574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-138f96ca-67e7-44ad-bab8-778e29cbacfb None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:08.024132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-138f96ca-67e7-44ad-bab8-778e29cbacfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:08.075575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e7baf3e-a447-42cf-a703-ebd626c2c03f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:08.113344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:08.114593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.115105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.115714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.117198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:08.119112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:08.121403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 87/167] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:07 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:08.122565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:08.156631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:08.177680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 81/168] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:08.178580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f96ca-67e7-44ad-bab8-778e29cbacfb None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:08.236569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a98d8b1-4db1-409a-a390-0894538940f1 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:08.260432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:08.275422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:08.276470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.276957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.277451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.278468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:08.278999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:08.280639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:08.290369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 88/169] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:08 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:08.293679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:08.358754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-138f96ca-67e7-44ad-bab8-778e29cbacfb None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:08.399176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:08.414014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ffbf2cf-63f2-40b2-86bc-4e1670077b76 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:08.435493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813dc040-567d-44af-8067-c0245ba10530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:08.436174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813dc040-567d-44af-8067-c0245ba10530 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.436464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813dc040-567d-44af-8067-c0245ba10530 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.436747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813dc040-567d-44af-8067-c0245ba10530 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.437666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813dc040-567d-44af-8067-c0245ba10530 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=984129f5bec74f43b4a666f392319525) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:08.437988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813dc040-567d-44af-8067-c0245ba10530 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:08.439617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813dc040-567d-44af-8067-c0245ba10530 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:08.446385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 82/170] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:08 2022] GET /identity/v3/role_assignments?user.id=984129f5bec74f43b4a666f392319525&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:08.447644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19989a4-f2c1-4e34-97c6-8ff3b3fc0252 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.447931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19989a4-f2c1-4e34-97c6-8ff3b3fc0252 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.448210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19989a4-f2c1-4e34-97c6-8ff3b3fc0252 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.452150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 83/171] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:08 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:08.452905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19989a4-f2c1-4e34-97c6-8ff3b3fc0252 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:08.466783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63fc8f17-c518-4b53-9169-d1a1b1c007d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:08.467640 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc8f17-c518-4b53-9169-d1a1b1c007d6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.467929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc8f17-c518-4b53-9169-d1a1b1c007d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.468211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63fc8f17-c518-4b53-9169-d1a1b1c007d6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.498508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63fc8f17-c518-4b53-9169-d1a1b1c007d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-63fc8f17-c518-4b53-9169-d1a1b1c007d6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:08.527259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:08.528193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:08.528961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 89/172] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:08.530705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb948e0d-db05-4590-9608-5935f4605f26 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.530705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb948e0d-db05-4590-9608-5935f4605f26 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.530705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb948e0d-db05-4590-9608-5935f4605f26 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.589300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-eb948e0d-db05-4590-9608-5935f4605f26 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:08.590584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-eb948e0d-db05-4590-9608-5935f4605f26 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:08.714034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19989a4-f2c1-4e34-97c6-8ff3b3fc0252 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:08.718217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 90/173] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:08.718881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb948e0d-db05-4590-9608-5935f4605f26 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:08.739225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:08.782819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:08.783636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.783920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.784197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.790241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:08.791814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:08.793577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 84/174] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:08 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 341 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:08.794348 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:08.872657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:08.904292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb948e0d-db05-4590-9608-5935f4605f26 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:08.950971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:08.951665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:08.951955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:08.952236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:08.953011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:08.953323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:08.955492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:08.968002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 91/175] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:08 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:08.968878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:08.982037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac92d9d5-5a9a-4526-87a3-a48ad3317904 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.034748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.036733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.037491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.038006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.049313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:09.058541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:09.059475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 85/176] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 266 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:09.063623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.115865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.117751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747fb2ce-25d1-46cc-bc14-fe193977b75f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.154292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079e3ec4-806d-48b7-9834-2b42c83515b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.154965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079e3ec4-806d-48b7-9834-2b42c83515b9 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.155246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079e3ec4-806d-48b7-9834-2b42c83515b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.155517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079e3ec4-806d-48b7-9834-2b42c83515b9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.171712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.172779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19770773-d0dc-4a21-b36a-3124789e1c3a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.209773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d11c7e2-fa74-431f-9719-9ae6fdedcaac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.210653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d11c7e2-fa74-431f-9719-9ae6fdedcaac None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.210995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d11c7e2-fa74-431f-9719-9ae6fdedcaac None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.211239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d11c7e2-fa74-431f-9719-9ae6fdedcaac None admin] PATH_INFO: `/v3/users/56dd0fcc7125401f8bc4841e48ca9167` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.212769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d11c7e2-fa74-431f-9719-9ae6fdedcaac None admin] RBAC: Authorizing `identity:get_user(user_id=56dd0fcc7125401f8bc4841e48ca9167)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:09.214386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d11c7e2-fa74-431f-9719-9ae6fdedcaac None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:09.216072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 86/177] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/users/56dd0fcc7125401f8bc4841e48ca9167 => generated 275 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:09.220789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7fb007-2194-4f94-92cf-8051428c8e3c None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.220789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7fb007-2194-4f94-92cf-8051428c8e3c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.220789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba7fb007-2194-4f94-92cf-8051428c8e3c None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.220789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 87/178] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:09 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:09.220789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7fb007-2194-4f94-92cf-8051428c8e3c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.235255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079e3ec4-806d-48b7-9834-2b42c83515b9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:09.236729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079e3ec4-806d-48b7-9834-2b42c83515b9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:09.331761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba7fb007-2194-4f94-92cf-8051428c8e3c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.340028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 92/179] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:08 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:09.340696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.341403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5ae3b-7a98-4e17-817a-772ddfe8dc4f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.341749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5ae3b-7a98-4e17-817a-772ddfe8dc4f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.341908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae5ae3b-7a98-4e17-817a-772ddfe8dc4f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.347198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 93/180] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:09.347198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae5ae3b-7a98-4e17-817a-772ddfe8dc4f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.375371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f91ca3e-7d79-4deb-8316-591625c9679a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.376038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91ca3e-7d79-4deb-8316-591625c9679a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.376316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91ca3e-7d79-4deb-8316-591625c9679a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.376584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f91ca3e-7d79-4deb-8316-591625c9679a None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.377324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f91ca3e-7d79-4deb-8316-591625c9679a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:09.377635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f91ca3e-7d79-4deb-8316-591625c9679a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:09.379043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f91ca3e-7d79-4deb-8316-591625c9679a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:09.388744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 88/181] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:09.394208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.394350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a60775-0d3c-4129-b383-b3891b97f537 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.394615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a60775-0d3c-4129-b383-b3891b97f537 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.394892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a60775-0d3c-4129-b383-b3891b97f537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.412589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-72a60775-0d3c-4129-b383-b3891b97f537 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:09.413768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-72a60775-0d3c-4129-b383-b3891b97f537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:09.452879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae5ae3b-7a98-4e17-817a-772ddfe8dc4f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.482425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 89/182] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:09.483301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a60775-0d3c-4129-b383-b3891b97f537 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.492309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715356cf-e7d2-48da-847d-324184655848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.493031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715356cf-e7d2-48da-847d-324184655848 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.493374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715356cf-e7d2-48da-847d-324184655848 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.493768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715356cf-e7d2-48da-847d-324184655848 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.561031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715356cf-e7d2-48da-847d-324184655848 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:09.562500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715356cf-e7d2-48da-847d-324184655848 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:09.592808 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.596653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a60775-0d3c-4129-b383-b3891b97f537 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.608975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.629922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbe88fa-e847-4245-acea-ea52a99b0e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.630794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe88fa-e847-4245-acea-ea52a99b0e9e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.631098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe88fa-e847-4245-acea-ea52a99b0e9e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.631376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbe88fa-e847-4245-acea-ea52a99b0e9e None admin] PATH_INFO: `/v3/projects/58077090af8b45c8a70613a6bc5245f8` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.632623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbe88fa-e847-4245-acea-ea52a99b0e9e None admin] RBAC: Authorizing `identity:get_project(project_id=58077090af8b45c8a70613a6bc5245f8)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:09.634280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbe88fa-e847-4245-acea-ea52a99b0e9e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:09.636105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 90/183] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/projects/58077090af8b45c8a70613a6bc5245f8 => generated 312 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:09.637266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed1756-bf78-4319-9eab-651fb337a41d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.637551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed1756-bf78-4319-9eab-651fb337a41d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.637832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ed1756-bf78-4319-9eab-651fb337a41d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.655235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-69ed1756-bf78-4319-9eab-651fb337a41d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:09.656424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-69ed1756-bf78-4319-9eab-651fb337a41d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:09.665997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 94/184] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:09.666987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715356cf-e7d2-48da-847d-324184655848 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.723191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 91/185] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:09.725298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94be6a24-9894-4d37-90ee-1a05b44104e6 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.725531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94be6a24-9894-4d37-90ee-1a05b44104e6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.725809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94be6a24-9894-4d37-90ee-1a05b44104e6 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.727609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 92/186] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:09.727816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94be6a24-9894-4d37-90ee-1a05b44104e6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.773596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715356cf-e7d2-48da-847d-324184655848 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.806883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.807749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.808026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.808298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] PATH_INFO: `/v3/users/3935d12d87124540908350a20c3b6af8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.823141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] RBAC: Authorizing `identity:get_user(user_id=3935d12d87124540908350a20c3b6af8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:09.824594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:09.826238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 95/187] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/users/3935d12d87124540908350a20c3b6af8 => generated 283 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:09.827021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.829518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.832044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:09.838904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94be6a24-9894-4d37-90ee-1a05b44104e6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.873170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.873915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.874197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.874474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.884866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:09.892653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:09.893648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:09.894537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 93/188] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:09.895370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.929716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0686a1de-c1ba-41f1-a97a-99ad21694a84 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:09.963352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:09.964001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:09.964286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:09.964562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:09.965392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=58077090af8b45c8a70613a6bc5245f8, user.id=56dd0fcc7125401f8bc4841e48ca9167) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:09.965702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:09.967172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:09.972807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 96/189] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/role_assignments?user.id=56dd0fcc7125401f8bc4841e48ca9167&scope.project.id=58077090af8b45c8a70613a6bc5245f8&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 608 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:09.987167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:09.999561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65522ef3-bf48-4efb-addd-7b5a08961fdf None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.041632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2436fd18-0795-4cdd-b290-3182c4a2e4c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.042516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2436fd18-0795-4cdd-b290-3182c4a2e4c9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.042817 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2436fd18-0795-4cdd-b290-3182c4a2e4c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.043095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2436fd18-0795-4cdd-b290-3182c4a2e4c9 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.044360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2436fd18-0795-4cdd-b290-3182c4a2e4c9 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.045869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2436fd18-0795-4cdd-b290-3182c4a2e4c9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.047676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 94/190] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.053106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534707da-adbc-4588-914f-c72d5beab4cd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.053106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534707da-adbc-4588-914f-c72d5beab4cd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.053106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534707da-adbc-4588-914f-c72d5beab4cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.069477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-534707da-adbc-4588-914f-c72d5beab4cd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:10.070723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-534707da-adbc-4588-914f-c72d5beab4cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:10.072126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.091338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.114970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d960cb-aed5-4710-a65d-5a5b71bd1a99 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.153378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 95/191] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:10.154965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.155781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.156061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.156338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.159683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534707da-adbc-4588-914f-c72d5beab4cd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.171550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:10.181368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:10.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:10.183345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:10.188256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 97/192] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:09 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:10.188256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.270812 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534707da-adbc-4588-914f-c72d5beab4cd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.290879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459734aa-c594-4bff-a9d9-1d51897729be None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.304363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.305009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.305290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.305561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.315105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.323593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.324258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.324541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.324816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.325604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] RBAC: Adding query filter params (name=reader) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:10.325911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.327354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.329419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.341031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 98/193] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.341031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.374752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.376249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.436533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8df784aa-a708-455f-b785-661afcacb65a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.470378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.471063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.471345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.471623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.472455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] RBAC: Adding query filter params (role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=56dd0fcc7125401f8bc4841e48ca9167) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:10.472770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.474628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 96/194] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:10.479611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.479726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.494794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 99/195] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/role_assignments?user.id=56dd0fcc7125401f8bc4841e48ca9167&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=1ff35caa6d284df39cd287349f66c567 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.494794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.538810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.540202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.589460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1d1d7c-0678-4461-8333-deddec038d47 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.592953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ceac25-f6e7-4952-ba3a-8daf86c941a8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.625417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.626234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.626520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.626817 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.628028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.630021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.631707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 97/196] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/domains/default => generated 199 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.632466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.632592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.633428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.633719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.634006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.635066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:10.635181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.636502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.647786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 100/197] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.647786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.756261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cfd54b-e8d4-4e47-90aa-e413e9d8e60b None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.761331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7779423-af1e-484d-9894-2cce3e121355 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.775722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.779349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:10.790587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.791495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.791778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.792053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] PATH_INFO: `/v3/users/25d5254a28dc4049951e1b7dfd25ccd2` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.793259 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] RBAC: Authorizing `identity:get_user(user_id=25d5254a28dc4049951e1b7dfd25ccd2)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.794718 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.794855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.795762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.796047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.796574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.796673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 98/198] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/users/25d5254a28dc4049951e1b7dfd25ccd2 => generated 289 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.797145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.797537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.799019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.800732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 101/199] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.801529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.897488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091c254a-d7cd-481e-bb72-cd8ba2d00113 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.899912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-679f4ab1-bd33-4f13-b9bc-4bfe63a658a7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:10.931138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.931807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.932091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.932229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:10.932659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.933043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:10.933143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.933326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:10.933601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] PATH_INFO: `/v3/users/3935d12d87124540908350a20c3b6af8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:10.934480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.934855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] RBAC: Authorizing `identity:get_user(user_id=3935d12d87124540908350a20c3b6af8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:10.936295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:10.938099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 102/200] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/users/3935d12d87124540908350a20c3b6af8 => generated 283 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:10.938686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:10.992343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:11.013566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 99/201] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:10 2022] POST /identity/v3/users => generated 297 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:11.014389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:11.044541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a82ca8-f1e0-4fb6-9c89-ef0abf724a64 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:11.072419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:11.084530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:11.085210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:11.085498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:11.085777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:11.086658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=3935d12d87124540908350a20c3b6af8) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:11.087011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:11.088505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:11.094566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 103/202] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:10 2022] GET /identity/v3/role_assignments?user.id=3935d12d87124540908350a20c3b6af8&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:11.095459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:11.153011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f63d5be-2e19-406c-80fb-f60cb7fe03da None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:11.202474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:11.203290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:11.203577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:11.203853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:11.272075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03c486-5086-4477-86f8-df00beaf14a2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:11.290746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:11.312135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:11.314293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:11.316932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:11.318487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:11.319421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:11.319723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:11.377235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:11.415098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:11.416637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:11.446845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 100/203] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:11.452006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:11.538644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 104/204] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 443 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:11.544598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:11.559111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260c2752-7bc1-40d9-a0fa-f0eff2669503 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:11.564136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:11.670227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:11.671208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:11.671494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:11.671774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:11.672987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:11.674460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:11.677515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 101/205] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:11 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:11.683293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:11.710793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:11.751098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209cbffb-96b7-4c3e-a3a1-47af67c53814 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:11.818437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:11.819701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:11.820179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:11.820645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:11.822340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:11.824037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:11.826791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 105/206] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:11 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:11.834546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:11.862350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94a0abd-99d7-42c2-aa6b-615f6ab390df None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:11.930726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0276a1d-db31-4b7b-ba9e-b9b4134eba15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:11.932489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0276a1d-db31-4b7b-ba9e-b9b4134eba15 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:11.933588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0276a1d-db31-4b7b-ba9e-b9b4134eba15 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:11.934215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0276a1d-db31-4b7b-ba9e-b9b4134eba15 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:11.936228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0276a1d-db31-4b7b-ba9e-b9b4134eba15 None admin] RBAC: Adding query filter params (role.id=ae7f6ead5e04485fa45d5b011096569d, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36, user.id=25d5254a28dc4049951e1b7dfd25ccd2) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:11.936544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0276a1d-db31-4b7b-ba9e-b9b4134eba15 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:11.938248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0276a1d-db31-4b7b-ba9e-b9b4134eba15 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:11.946306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 102/207] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:11 2022] GET /identity/v3/role_assignments?user.id=25d5254a28dc4049951e1b7dfd25ccd2&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=ae7f6ead5e04485fa45d5b011096569d => generated 608 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:11.980247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:12.007294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c7ad4f-0092-445c-b9dd-7c1b664dd207 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:12.064620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c245e82f-d802-44f3-a789-fbe6fa584d3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:12.065307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245e82f-d802-44f3-a789-fbe6fa584d3a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:12.065579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245e82f-d802-44f3-a789-fbe6fa584d3a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:12.065853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c245e82f-d802-44f3-a789-fbe6fa584d3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:12.066713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c245e82f-d802-44f3-a789-fbe6fa584d3a None admin] RBAC: Adding query filter params (role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=3935d12d87124540908350a20c3b6af8) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:12.067583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c245e82f-d802-44f3-a789-fbe6fa584d3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:12.069316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c245e82f-d802-44f3-a789-fbe6fa584d3a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:12.079639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 106/208] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:11 2022] GET /identity/v3/role_assignments?user.id=3935d12d87124540908350a20c3b6af8&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=1ff35caa6d284df39cd287349f66c567 => generated 259 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:13.318811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9139d63-2ba0-496d-9d83-ad1e9ebf5f18 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.319863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9139d63-2ba0-496d-9d83-ad1e9ebf5f18 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.320346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9139d63-2ba0-496d-9d83-ad1e9ebf5f18 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.322199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 103/209] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:13 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:13.327124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57910718-9441-464b-a3f2-5d8ecf6d4735 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.327520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57910718-9441-464b-a3f2-5d8ecf6d4735 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.327703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57910718-9441-464b-a3f2-5d8ecf6d4735 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.345546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-57910718-9441-464b-a3f2-5d8ecf6d4735 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:13.346733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-57910718-9441-464b-a3f2-5d8ecf6d4735 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:13.390675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd38855-3861-4713-b750-4e21850c1c80 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.391000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd38855-3861-4713-b750-4e21850c1c80 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.391286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd38855-3861-4713-b750-4e21850c1c80 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.392834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 104/210] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:13.398543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ed332-ee3a-4544-88ef-047648766f49 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.399127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ed332-ee3a-4544-88ef-047648766f49 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.399406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-588ed332-ee3a-4544-88ef-047648766f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.416184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-588ed332-ee3a-4544-88ef-047648766f49 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:13.417379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-588ed332-ee3a-4544-88ef-047648766f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:13.421422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 107/211] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:13.443809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57910718-9441-464b-a3f2-5d8ecf6d4735 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:13.487645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 105/212] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:13.511645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ed332-ee3a-4544-88ef-047648766f49 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:13.527127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:13.527398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:13.547496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57910718-9441-464b-a3f2-5d8ecf6d4735 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:13.585417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:13.586372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.586663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.586964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.600556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:13.608821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:13.609684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:13.610906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 108/213] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:13.616917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:13.637398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-588ed332-ee3a-4544-88ef-047648766f49 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:13.673787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e6dd32c-1d1a-4137-a94e-bd7d69bd6652 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:13.674596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6dd32c-1d1a-4137-a94e-bd7d69bd6652 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.674909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6dd32c-1d1a-4137-a94e-bd7d69bd6652 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.675200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6dd32c-1d1a-4137-a94e-bd7d69bd6652 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.685696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e6dd32c-1d1a-4137-a94e-bd7d69bd6652 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-5e6dd32c-1d1a-4137-a94e-bd7d69bd6652 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:13.694132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:13.695056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:13.696075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 106/214] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:13 2022] GET /identity/v3/roles/member => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:13.696987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ce3ae-5197-499f-abe2-253db761123c None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.696987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ce3ae-5197-499f-abe2-253db761123c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.697237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3ce3ae-5197-499f-abe2-253db761123c None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.701587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 107/215] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:13.701587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3ce3ae-5197-499f-abe2-253db761123c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:13.735883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ceba8a1-1a5d-4391-999b-2b2fef3ecff2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:13.774275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36814abc-50ef-4e93-a4b6-e42eaa834edb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:13.774987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36814abc-50ef-4e93-a4b6-e42eaa834edb None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.775274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36814abc-50ef-4e93-a4b6-e42eaa834edb None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.775556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36814abc-50ef-4e93-a4b6-e42eaa834edb None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.776315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36814abc-50ef-4e93-a4b6-e42eaa834edb None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:13.776622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36814abc-50ef-4e93-a4b6-e42eaa834edb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:13.778166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36814abc-50ef-4e93-a4b6-e42eaa834edb None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:13.780600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:13.781229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:13.788011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 109/216] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:13 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:13.789085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2d2a9e-282f-4954-827a-794365823660 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.789376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2d2a9e-282f-4954-827a-794365823660 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.789658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2d2a9e-282f-4954-827a-794365823660 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.808244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0f2d2a9e-282f-4954-827a-794365823660 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:13.809467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0f2d2a9e-282f-4954-827a-794365823660 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:13.822945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3ce3ae-5197-499f-abe2-253db761123c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:13.858796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:13.859400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:13.859686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:13.859966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:13.860720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:13.861033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:13.862409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:13.871144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 108/217] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:13.876114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:13.884838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 110/218] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:13.886017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2d2a9e-282f-4954-827a-794365823660 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:13.992650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc3ad6-ad0f-4f10-b119-cf9d5f862a97 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:13.999238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2d2a9e-282f-4954-827a-794365823660 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.023956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:14.028311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:14.030544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.031465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.031765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.032068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] PATH_INFO: `/v3/users/3935d12d87124540908350a20c3b6af8` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.036391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] RBAC: Authorizing `identity:get_user(user_id=3935d12d87124540908350a20c3b6af8)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.036391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.036681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.041612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 109/219] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:13 2022] GET /identity/v3/users/3935d12d87124540908350a20c3b6af8 => generated 283 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:14.041731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.041826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.041901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.041901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] PATH_INFO: `/v3/users/112eb9a1b66347948b127628845aff42` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.055218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] RBAC: Authorizing `identity:get_user(user_id=112eb9a1b66347948b127628845aff42)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.056733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.058831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 111/220] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:13 2022] GET /identity/v3/users/112eb9a1b66347948b127628845aff42 => generated 297 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:14.059658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.154702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8a5a23e-6dbc-43e3-95dd-d5b5b8bb7155 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.161606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732ccad2-ad6c-4c1c-a589-d3693086d1d4 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.189347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.190141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.190426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.190702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.191930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.193557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.194965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80cb743f-f014-4ae2-a2b8-e208ec3b4e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.195219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 110/221] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:14.195830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cb743f-f014-4ae2-a2b8-e208ec3b4e9b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.195987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.196158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cb743f-f014-4ae2-a2b8-e208ec3b4e9b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.196388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80cb743f-f014-4ae2-a2b8-e208ec3b4e9b None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.197559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cb743f-f014-4ae2-a2b8-e208ec3b4e9b None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.199027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80cb743f-f014-4ae2-a2b8-e208ec3b4e9b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.200715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 112/222] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:14 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:14.204079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec2e9a-9f3b-4e7d-aaf8-afe709ee148a None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.204367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec2e9a-9f3b-4e7d-aaf8-afe709ee148a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.204642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30ec2e9a-9f3b-4e7d-aaf8-afe709ee148a None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.205927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 113/223] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:14.206608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ec2e9a-9f3b-4e7d-aaf8-afe709ee148a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.262441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:14.262615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:14.296552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed66e7b8-c159-4b6a-a404-6a3f520c3563 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.307285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30ec2e9a-9f3b-4e7d-aaf8-afe709ee148a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.329695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e9d40a5-cf97-439a-b159-a39ff0b78f5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.330351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d40a5-cf97-439a-b159-a39ff0b78f5c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.330638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d40a5-cf97-439a-b159-a39ff0b78f5c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.330948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e9d40a5-cf97-439a-b159-a39ff0b78f5c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.340909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.341578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.341855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.342126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.342862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.344303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.398967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9d40a5-cf97-439a-b159-a39ff0b78f5c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.400477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e9d40a5-cf97-439a-b159-a39ff0b78f5c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.428218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 114/224] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:14 2022] POST /identity/v3/users => generated 297 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:14.428218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.482409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:14.526822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 111/225] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:14.526822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd64c07-44ae-43d2-9b8d-2960b0fe43d3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.526822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd64c07-44ae-43d2-9b8d-2960b0fe43d3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.526822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd64c07-44ae-43d2-9b8d-2960b0fe43d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.542414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e037b8-7ccc-446b-99b6-c0cf8b9bb8b0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.543154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ccd64c07-44ae-43d2-9b8d-2960b0fe43d3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:14.544382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ccd64c07-44ae-43d2-9b8d-2960b0fe43d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:14.555502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:14.580941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.583893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.583893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.583893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/3935d12d87124540908350a20c3b6af8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.585548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] RBAC: Authorizing `identity:create_grant(project_id=5586dcebfe8c4e81bb952f00a518722e, user_id=3935d12d87124540908350a20c3b6af8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.587051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.624135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 112/226] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:14.624827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd64c07-44ae-43d2-9b8d-2960b0fe43d3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.663659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 115/227] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:14 2022] PUT /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/3935d12d87124540908350a20c3b6af8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:14.664521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.760455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:14.764174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd64c07-44ae-43d2-9b8d-2960b0fe43d3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.813722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.815163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.815660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.816166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.817976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:14.819731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:14.820858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceaef1eb-eaa2-4051-9144-472df1f7aa3b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:14.822486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 113/228] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:14 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:14.828908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.860052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:14.860860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:14.861146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:14.861421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:14.874176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:14.882428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:14.883371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:14.884193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 116/229] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:14 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:14.890006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:14.930235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:15.012363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cccc5d4-47ec-4084-a610-89488283bf62 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:15.013061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cc9f45-163b-417f-ade1-47436d06deb8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:15.053239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:15.053918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:15.054200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:15.054482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:15.055864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:15.056177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:15.057585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:15.068555 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 117/230] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:14 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:15.073602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:15.091254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:15.091972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:15.092253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:15.092517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:15.093407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=112eb9a1b66347948b127628845aff42) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:15.093718 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:15.098807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:15.110326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 114/231] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:14 2022] GET /identity/v3/role_assignments?user.id=112eb9a1b66347948b127628845aff42&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 259 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:15.192180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:15.201880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc996c21-1bd0-46b2-9655-4bbf04375212 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:15.242216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:15.243143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:15.243425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:15.243704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] PATH_INFO: `/v3/users/3935d12d87124540908350a20c3b6af8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:15.245023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] RBAC: Authorizing `identity:get_user(user_id=3935d12d87124540908350a20c3b6af8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:15.246563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:15.248304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 118/232] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:15 2022] GET /identity/v3/users/3935d12d87124540908350a20c3b6af8 => generated 283 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:15.253951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:15.287529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:15.365662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeb2d50-55b1-4b17-a6f8-ca95dc82ce93 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:15.403999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c8f07e-68cf-4739-b255-1b533e618026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:15.404887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8f07e-68cf-4739-b255-1b533e618026 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:15.405168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8f07e-68cf-4739-b255-1b533e618026 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:15.405439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c8f07e-68cf-4739-b255-1b533e618026 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:15.406641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c8f07e-68cf-4739-b255-1b533e618026 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:15.408133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c8f07e-68cf-4739-b255-1b533e618026 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:15.410134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 115/233] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:15 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:15.415999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:15.520603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0501e7b-04d7-40ac-aa7d-fddf31102bc8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:15.561959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a481ec3b-1340-414d-a27e-74b77c2db3d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:15.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a481ec3b-1340-414d-a27e-74b77c2db3d6 None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:15.563147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a481ec3b-1340-414d-a27e-74b77c2db3d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:15.563403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a481ec3b-1340-414d-a27e-74b77c2db3d6 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/3935d12d87124540908350a20c3b6af8/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:15.568915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a481ec3b-1340-414d-a27e-74b77c2db3d6 None admin] RBAC: Authorizing `identity:create_grant(project_id=5586dcebfe8c4e81bb952f00a518722e, user_id=3935d12d87124540908350a20c3b6af8, role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:15.570364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a481ec3b-1340-414d-a27e-74b77c2db3d6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:15.705511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 119/234] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:15 2022] PUT /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/3935d12d87124540908350a20c3b6af8/roles/1ff35caa6d284df39cd287349f66c567 => generated 0 bytes in 290 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:17.057287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7639c-d36a-4091-94e6-b9b5d57d7e4f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.057979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7639c-d36a-4091-94e6-b9b5d57d7e4f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.057979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de7639c-d36a-4091-94e6-b9b5d57d7e4f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.059593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 116/235] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:17.064783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1724219-ec6a-4250-9d18-ae64500e7cfa None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.065268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1724219-ec6a-4250-9d18-ae64500e7cfa None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.065523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1724219-ec6a-4250-9d18-ae64500e7cfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.082249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c1724219-ec6a-4250-9d18-ae64500e7cfa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:17.083980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c1724219-ec6a-4250-9d18-ae64500e7cfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:17.106616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:17.135935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd598e2-6166-4366-b449-2e271e93e1ef None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.136351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd598e2-6166-4366-b449-2e271e93e1ef None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.136629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd598e2-6166-4366-b449-2e271e93e1ef None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.138013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 117/236] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:17.143095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d168325f-853a-44d8-9709-3cf211f4b10f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.143378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d168325f-853a-44d8-9709-3cf211f4b10f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.143656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d168325f-853a-44d8-9709-3cf211f4b10f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.171852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d168325f-853a-44d8-9709-3cf211f4b10f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:17.173147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d168325f-853a-44d8-9709-3cf211f4b10f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:17.173934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 120/237] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:17.210023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1724219-ec6a-4250-9d18-ae64500e7cfa None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:17.314658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 118/238] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:17.320409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1724219-ec6a-4250-9d18-ae64500e7cfa None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:17.349621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d168325f-853a-44d8-9709-3cf211f4b10f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:17.359198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032a3548-c999-41e5-83a5-206d5fbbe493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:17.360007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a3548-c999-41e5-83a5-206d5fbbe493 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.360349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a3548-c999-41e5-83a5-206d5fbbe493 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.360628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032a3548-c999-41e5-83a5-206d5fbbe493 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.373167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032a3548-c999-41e5-83a5-206d5fbbe493 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-032a3548-c999-41e5-83a5-206d5fbbe493 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:17.381917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:17.382827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:17.383624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 121/239] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:17.387567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3846388f-4feb-481e-bd1d-a37ce1e4b64a None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.387992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3846388f-4feb-481e-bd1d-a37ce1e4b64a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.388279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3846388f-4feb-481e-bd1d-a37ce1e4b64a None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.390185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 122/240] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:17.393842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3846388f-4feb-481e-bd1d-a37ce1e4b64a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:17.403824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:17.408891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:17.480149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d168325f-853a-44d8-9709-3cf211f4b10f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:17.517197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e009bb90-6cb3-4627-ab6c-1a9c68d6b1dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:17.518330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e009bb90-6cb3-4627-ab6c-1a9c68d6b1dc None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.518857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e009bb90-6cb3-4627-ab6c-1a9c68d6b1dc None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.519305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e009bb90-6cb3-4627-ab6c-1a9c68d6b1dc None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.521386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e009bb90-6cb3-4627-ab6c-1a9c68d6b1dc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:17.523227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e009bb90-6cb3-4627-ab6c-1a9c68d6b1dc None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:17.525289 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 119/241] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:17 2022] GET /identity/v3/domains/default => generated 199 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:17.529681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3846388f-4feb-481e-bd1d-a37ce1e4b64a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:17.531294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523d0262-da30-4896-ae59-e6a0c391f1a7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.531762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523d0262-da30-4896-ae59-e6a0c391f1a7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.532215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-523d0262-da30-4896-ae59-e6a0c391f1a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.549137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-523d0262-da30-4896-ae59-e6a0c391f1a7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:17.551262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-523d0262-da30-4896-ae59-e6a0c391f1a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:17.571896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:17.572886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.573236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.573586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.574701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:17.575517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:17.577054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:17.589863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 123/242] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:17 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:17.590900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:17.638842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 120/243] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:17.638842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523d0262-da30-4896-ae59-e6a0c391f1a7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:17.665248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:17.675633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:17.706557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e9b893-7051-4265-bb2f-bbef074e7eb8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:17.743470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e034084f-4d1d-4863-ace7-8a53c9057028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:17.744150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034084f-4d1d-4863-ace7-8a53c9057028 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.744440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034084f-4d1d-4863-ace7-8a53c9057028 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.744717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e034084f-4d1d-4863-ace7-8a53c9057028 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.745436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e034084f-4d1d-4863-ace7-8a53c9057028 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:17.746929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e034084f-4d1d-4863-ace7-8a53c9057028 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:17.754064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-523d0262-da30-4896-ae59-e6a0c391f1a7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:17.790032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:17.790931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.791209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.791478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] PATH_INFO: `/v3/users/fe13d9b013b342cab05d11a5bc81fd45` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.805536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] RBAC: Authorizing `identity:get_user(user_id=fe13d9b013b342cab05d11a5bc81fd45)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:17.807265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:17.814992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 124/244] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:17 2022] POST /identity/v3/users => generated 293 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:17.815183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 121/245] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:17 2022] GET /identity/v3/users/fe13d9b013b342cab05d11a5bc81fd45 => generated 297 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:17.815183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:17.815360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63429dcf-ed8b-458b-afa2-e5abd33637c0 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:17.815360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63429dcf-ed8b-458b-afa2-e5abd33637c0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:17.815506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63429dcf-ed8b-458b-afa2-e5abd33637c0 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:17.822875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 125/246] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:17 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:17.822875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63429dcf-ed8b-458b-afa2-e5abd33637c0 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:17.944789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:17.960777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63429dcf-ed8b-458b-afa2-e5abd33637c0 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:17.970740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cecb45c-893c-4a6a-b514-9246a0dd39ac None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:17.995036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:18.002107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.002921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.005102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.005102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.018135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7244e7-9b3e-4b77-9431-ba87eb4c6077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.019097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7244e7-9b3e-4b77-9431-ba87eb4c6077 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.019443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7244e7-9b3e-4b77-9431-ba87eb4c6077 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.019722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7244e7-9b3e-4b77-9431-ba87eb4c6077 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.033547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7244e7-9b3e-4b77-9431-ba87eb4c6077 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-4a7244e7-9b3e-4b77-9431-ba87eb4c6077 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:18.042122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:18.043285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:18.044249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 122/247] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:18.045020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e8546-87ef-46ce-a223-c544e22a6efa None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.045087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e8546-87ef-46ce-a223-c544e22a6efa None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.045334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e8546-87ef-46ce-a223-c544e22a6efa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.065914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d88e8546-87ef-46ce-a223-c544e22a6efa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:18.067284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d88e8546-87ef-46ce-a223-c544e22a6efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:18.103822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:18.105524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:18.170867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 123/248] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:18.172084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e8546-87ef-46ce-a223-c544e22a6efa None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.236030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 126/249] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:18.241333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.275653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:18.315839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e8546-87ef-46ce-a223-c544e22a6efa None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:18.319473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:18.356314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.356910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.357192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.357463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.358319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:18.358626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:18.360197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:18.365647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f04e32-d1ec-45aa-b365-f8b7595596b6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:18.369440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 124/250] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:18.370223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.405807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.407131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.407738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.408274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.420524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:18.429404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:18.430308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:18.431142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:18.434117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 127/251] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:18.435213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.484251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aaae331-dd1a-476c-be51-bc7710787e03 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:18.522061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e6b746-dab6-469f-ae55-e3d1b5e4834b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.522924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6b746-dab6-469f-ae55-e3d1b5e4834b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.523204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6b746-dab6-469f-ae55-e3d1b5e4834b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.523477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e6b746-dab6-469f-ae55-e3d1b5e4834b None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.524963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e6b746-dab6-469f-ae55-e3d1b5e4834b None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:18.526527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e6b746-dab6-469f-ae55-e3d1b5e4834b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:18.528394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 125/252] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:18.534882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10f3bf-b6b6-42b2-ab17-70b0180ef004 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.535185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10f3bf-b6b6-42b2-ab17-70b0180ef004 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.536163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10f3bf-b6b6-42b2-ab17-70b0180ef004 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.538118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 126/253] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:18 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:18.538828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10f3bf-b6b6-42b2-ab17-70b0180ef004 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.540053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:18.543157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:18.554580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c01519-f52b-422b-b63e-725476fdff5d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:18.597294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.598937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.599317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.599710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] PATH_INFO: `/v3/users/3935d12d87124540908350a20c3b6af8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.601034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] RBAC: Authorizing `identity:get_user(user_id=3935d12d87124540908350a20c3b6af8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:18.602607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:18.604380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 128/254] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/users/3935d12d87124540908350a20c3b6af8 => generated 283 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:18.605245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.655530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10f3bf-b6b6-42b2-ab17-70b0180ef004 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:18.692470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df8d7ac-bace-4c1c-9f31-07c5ce37b48e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.693461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df8d7ac-bace-4c1c-9f31-07c5ce37b48e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.693954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df8d7ac-bace-4c1c-9f31-07c5ce37b48e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.694400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df8d7ac-bace-4c1c-9f31-07c5ce37b48e None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.695489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df8d7ac-bace-4c1c-9f31-07c5ce37b48e None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:18.696014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df8d7ac-bace-4c1c-9f31-07c5ce37b48e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:18.697608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5df8d7ac-bace-4c1c-9f31-07c5ce37b48e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:18.708096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 127/255] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:18.709806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f722e4-fdf9-40fc-866d-5f86fd43e6cb None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.710319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f722e4-fdf9-40fc-866d-5f86fd43e6cb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.710787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f722e4-fdf9-40fc-866d-5f86fd43e6cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.715609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b8431-56fd-4925-914b-4bd73eca8680 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:18.737191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-61f722e4-fdf9-40fc-866d-5f86fd43e6cb None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:18.739754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-61f722e4-fdf9-40fc-866d-5f86fd43e6cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:18.760721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.762045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.762498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.762972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:18.764005 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] RBAC: Adding query filter params (role.id=ae7f6ead5e04485fa45d5b011096569d, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=fe13d9b013b342cab05d11a5bc81fd45) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:18.764487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:18.766932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:18.775512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 129/256] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/role_assignments?user.id=fe13d9b013b342cab05d11a5bc81fd45&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=ae7f6ead5e04485fa45d5b011096569d => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:18.776590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.803802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:18.836694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:18.851163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 128/257] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:18.852178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f722e4-fdf9-40fc-866d-5f86fd43e6cb None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:18.906180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfc97e76-542b-44b3-865e-8bf46554dcb2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:18.953558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:18.954580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:18.955064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:18.955508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:19.064036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f722e4-fdf9-40fc-866d-5f86fd43e6cb None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:19.069944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:19.071671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:19.097100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:19.164590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:19.165363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:19.165648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:19.165923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] PATH_INFO: `/v3/users/112eb9a1b66347948b127628845aff42` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:19.167229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] RBAC: Authorizing `identity:get_user(user_id=112eb9a1b66347948b127628845aff42)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:19.168629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:19.170469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 129/258] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/users/112eb9a1b66347948b127628845aff42 => generated 297 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:19.175155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:19.191119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 130/259] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 415 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:19.192820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:19.315247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-661a74e2-d046-47ee-8914-cc33686b2191 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:19.328520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:19.341251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:19.358002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:19.359210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:19.359687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:19.360167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:19.367601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:19.369327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:19.371774 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 131/260] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:19 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:19.373604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:19.404745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf7fb6d-2079-4e3a-88f9-94640e2e8804 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:19.443283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:19.444465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:19.444962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:19.445430 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:19.456157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:19.465341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:19.466290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:19.467464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:19.469998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 130/261] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:19.471081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:19.578002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0820151-a427-4ced-91f7-ab16484a5ce2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:19.616379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:19.617618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:19.618119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:19.618634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/112eb9a1b66347948b127628845aff42/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:19.620772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf5cb6b-b2b8-43c2-820d-2369121dca4c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:19.622962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:19.623903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] RBAC: Authorizing `identity:create_grant(project_id=5586dcebfe8c4e81bb952f00a518722e, user_id=112eb9a1b66347948b127628845aff42, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:19.625674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:19.667577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:19.668831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:19.669344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:19.669900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:19.671406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:19.673234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:19.675284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 132/262] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:19 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 302 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:19.677222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:19.700472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 131/263] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:19 2022] PUT /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/112eb9a1b66347948b127628845aff42/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:19.701904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:19.784186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:19.832017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94da72ec-0765-474c-a022-1e8a010e98ab None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:19.869873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:19.870640 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:19.870951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:19.871226 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:19.871989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:19.872295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:19.873726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:19.883442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 133/264] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:19.888807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:19.957046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d960181f-e459-4534-b66e-16c4ba7bac2a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:19.998547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37eaabb-63a7-451e-a9a3-f1c5b058c029 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:20.029270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:20.041653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:20.042593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.042900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.043280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] PATH_INFO: `/v3/users/3935d12d87124540908350a20c3b6af8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.044584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] RBAC: Authorizing `identity:get_user(user_id=3935d12d87124540908350a20c3b6af8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:20.046358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:20.048860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 134/265] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:19 2022] GET /identity/v3/users/3935d12d87124540908350a20c3b6af8 => generated 283 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:20.055641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:20.062285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:20.063001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.063287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.063565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.064487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=3935d12d87124540908350a20c3b6af8) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:20.064803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:20.066268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:20.085673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 132/266] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:19 2022] GET /identity/v3/role_assignments?user.id=3935d12d87124540908350a20c3b6af8&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 384 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:20.164976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c68d557-f9f7-404f-be93-6335c615b219 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:20.200989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:20.201775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.201980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.202251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.263521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:20.278378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:20.279947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:20.390148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 135/267] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 14 11:59:20.395977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:20.447720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:20.504496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7956a6-6292-4bc9-835c-df58e67aecf7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:20.540242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39aa79f-626f-4b0c-b8a2-8b4a37a08b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:20.542107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39aa79f-626f-4b0c-b8a2-8b4a37a08b15 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.542393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39aa79f-626f-4b0c-b8a2-8b4a37a08b15 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.542666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39aa79f-626f-4b0c-b8a2-8b4a37a08b15 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.543962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39aa79f-626f-4b0c-b8a2-8b4a37a08b15 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:20.545412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39aa79f-626f-4b0c-b8a2-8b4a37a08b15 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:20.549994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 133/268] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:20 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:20.554350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:20.568231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7e9e68-d16b-49c1-9c9a-5f24ab34436c None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.568526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7e9e68-d16b-49c1-9c9a-5f24ab34436c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.568776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7e9e68-d16b-49c1-9c9a-5f24ab34436c None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.573030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 134/269] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:20 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:20.582699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c32dee1-8b8e-4145-b6bb-6984a8216770 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.582699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c32dee1-8b8e-4145-b6bb-6984a8216770 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.582699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c32dee1-8b8e-4145-b6bb-6984a8216770 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.604477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2c32dee1-8b8e-4145-b6bb-6984a8216770 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:20.605677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2c32dee1-8b8e-4145-b6bb-6984a8216770 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:20.683139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:20.693374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 135/270] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:20.694791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d47e858-0526-461b-98ed-6a3d7b062bb3 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.695092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d47e858-0526-461b-98ed-6a3d7b062bb3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.695368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d47e858-0526-461b-98ed-6a3d7b062bb3 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.696665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 136/271] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:20.697145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56e4431-2232-41be-afc9-6b7c6f26a544 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:20.703102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a32f27c-d388-4737-9a7d-2e2b8247c88e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.703456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a32f27c-d388-4737-9a7d-2e2b8247c88e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.703792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a32f27c-d388-4737-9a7d-2e2b8247c88e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.734787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4a32f27c-d388-4737-9a7d-2e2b8247c88e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:20.734787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4a32f27c-d388-4737-9a7d-2e2b8247c88e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:20.757829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:20.758489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:20.758790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:20.759893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:20.760824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] RBAC: Adding query filter params (role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=3935d12d87124540908350a20c3b6af8) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:20.761141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:20.762712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:20.769461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 136/272] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:20 2022] GET /identity/v3/role_assignments?user.id=3935d12d87124540908350a20c3b6af8&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=1ff35caa6d284df39cd287349f66c567 => generated 608 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:20.770859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:20.791468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:20.819455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 137/273] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:20.847911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a32f27c-d388-4737-9a7d-2e2b8247c88e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:20.974793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a32f27c-d388-4737-9a7d-2e2b8247c88e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:21.011741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:21.012960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.013247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.013522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.031414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2262c4ea-6181-4826-8ed7-b561bffe82f7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:21.031713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:21.042836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:21.043721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:21.044512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:21.045107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 138/274] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:21.052662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:21.082250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:21.083730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.084018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.084410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.085977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:21.088142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:21.090554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 137/275] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:20 2022] GET /identity/v3/domains/default => generated 199 bytes in 321 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:21.096033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:21.098971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:21.120847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:21.183260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799b8bd-0b61-4149-8463-15f8ea144424 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:21.225339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:21.226021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.226304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.226582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.227360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:21.227668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:21.229315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:21.232678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0db7e8f8-c8bd-4332-b413-caa6584ba028 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:21.241112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 139/276] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:21.246636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:21.279829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:21.280968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.281388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.281782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.282509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:21.284241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:21.358488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:21.364416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce23a945-840e-4865-addb-31ec4c814860 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:21.389461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 138/277] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:21 2022] POST /identity/v3/users => generated 293 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:21.411797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd23b0e6-1fda-4637-a19c-f1546cd18a09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:21.412781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd23b0e6-1fda-4637-a19c-f1546cd18a09 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.413068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd23b0e6-1fda-4637-a19c-f1546cd18a09 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.413346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd23b0e6-1fda-4637-a19c-f1546cd18a09 None admin] PATH_INFO: `/v3/users/c41a13a77bd545818553456ae375acf7` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.430405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd23b0e6-1fda-4637-a19c-f1546cd18a09 None admin] RBAC: Authorizing `identity:get_user(user_id=c41a13a77bd545818553456ae375acf7)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:21.431949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd23b0e6-1fda-4637-a19c-f1546cd18a09 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:21.433749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 140/278] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:21 2022] GET /identity/v3/users/c41a13a77bd545818553456ae375acf7 => generated 293 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:21.439655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:21.490895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919d846e-8f1c-497d-916f-d4f3460c95c3 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.491627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919d846e-8f1c-497d-916f-d4f3460c95c3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.492139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-919d846e-8f1c-497d-916f-d4f3460c95c3 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.494356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 141/279] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:21.500027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504136fd-7cfe-4831-83a2-ce739ca23854 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.500317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504136fd-7cfe-4831-83a2-ce739ca23854 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.500599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504136fd-7cfe-4831-83a2-ce739ca23854 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.522136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-504136fd-7cfe-4831-83a2-ce739ca23854 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:21.523706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-504136fd-7cfe-4831-83a2-ce739ca23854 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:21.533634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:21.563200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4933e9b2-5a3a-453a-b693-a94252716a22 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:21.617007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:21.618814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.619121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.619999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.620239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, user.id=c41a13a77bd545818553456ae375acf7, scope.system=all) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:21.620549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:21.622071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:21.627825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 139/280] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Jul 14 11:59:21 2022] GET /identity/v3/role_assignments?user.id=c41a13a77bd545818553456ae375acf7&scope.system=all&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:21.633518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 142/281] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:21.657094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:21.789547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee2c9bd-ca9c-4693-a566-dbf45d843f15 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:21.827400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:21.828198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:21.828479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:21.828752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:21.836888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:21.841017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:21.849491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:21.850385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:21.851330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 140/282] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:21 2022] GET /identity/v3/roles/reader => generated 84 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:21.857997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504136fd-7cfe-4831-83a2-ce739ca23854 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:21.963930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:21.982363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504136fd-7cfe-4831-83a2-ce739ca23854 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:22.030999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:22.031687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.031973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.032247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.032995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:22.033300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:22.034689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:22.050295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 143/283] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:22.056630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:22.209661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a150ba-d96f-4db9-b10e-fb97705c18c8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:22.254649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9c6a54-e101-47f5-a6c6-a9bad44023a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:22.255535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c6a54-e101-47f5-a6c6-a9bad44023a0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.255728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c6a54-e101-47f5-a6c6-a9bad44023a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.256008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9c6a54-e101-47f5-a6c6-a9bad44023a0 None admin] PATH_INFO: `/v3/users/fe13d9b013b342cab05d11a5bc81fd45` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.257228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9c6a54-e101-47f5-a6c6-a9bad44023a0 None admin] RBAC: Authorizing `identity:get_user(user_id=fe13d9b013b342cab05d11a5bc81fd45)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:22.258695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9c6a54-e101-47f5-a6c6-a9bad44023a0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:22.260673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 141/284] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:22 2022] GET /identity/v3/users/fe13d9b013b342cab05d11a5bc81fd45 => generated 297 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:22.271894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:22.381469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ab69a-3003-409c-96dc-298232ca513f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.382115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ab69a-3003-409c-96dc-298232ca513f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.382509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44ab69a-3003-409c-96dc-298232ca513f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.384369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 142/285] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:22.391217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a066123-da84-430f-91ec-6499cb0759ce None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.391677 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a066123-da84-430f-91ec-6499cb0759ce None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.392111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a066123-da84-430f-91ec-6499cb0759ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.393654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fc7807-2c48-47e8-a226-64f4b0ee7443 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:22.423800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-7a066123-da84-430f-91ec-6499cb0759ce None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:22.425009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7a066123-da84-430f-91ec-6499cb0759ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:22.459497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:22.460379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.460665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.461007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.462238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:22.463729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:22.465738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 144/286] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:22 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:22.471213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:22.499630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:22.502279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 143/287] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:22.525060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:22.527385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a066123-da84-430f-91ec-6499cb0759ce None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:22.635695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a066123-da84-430f-91ec-6499cb0759ce None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:22.672420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:22.673229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.673593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.673793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.685061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:22.694450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:22.695366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:22.696171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:22.697429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 144/288] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:22.702230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:22.710706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c0f1c2-9529-415d-b10c-ab3b0203fd2c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:22.779209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:22.799632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:22.800602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.800914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.801759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/fe13d9b013b342cab05d11a5bc81fd45/roles/ae7f6ead5e04485fa45d5b011096569d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.808354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] RBAC: Authorizing `identity:create_grant(project_id=5586dcebfe8c4e81bb952f00a518722e, user_id=fe13d9b013b342cab05d11a5bc81fd45, role_id=ae7f6ead5e04485fa45d5b011096569d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:22.809822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:22.834335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3418656c-f6b0-48f1-89f5-afe16dde467e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:22.871404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:22.872535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:22.873024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:22.873501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:22.874460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:22.874997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:22.876612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:22.885792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 145/290] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:22 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:22.890429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 145/290] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:22 2022] PUT /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e/users/fe13d9b013b342cab05d11a5bc81fd45/roles/ae7f6ead5e04485fa45d5b011096569d => generated 0 bytes in 415 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:22.892244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:23.006028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82b5a08-6ea8-4791-913e-c0822329f0f4 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:23.025929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:23.044727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:23.045938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:23.046466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:23.046968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] PATH_INFO: `/v3/users/112eb9a1b66347948b127628845aff42` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:23.048419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] RBAC: Authorizing `identity:get_user(user_id=112eb9a1b66347948b127628845aff42)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:23.050083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:23.051991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 146/291] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:22 2022] GET /identity/v3/users/112eb9a1b66347948b127628845aff42 => generated 297 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:23.057855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:23.162607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:23.188669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c1f9861-66cf-4d5e-bd6b-5ca951babd5f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:23.227306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:23.227965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:23.228245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:23.228519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:23.326312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:23.327705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:23.421361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:23.452863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 146/292] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:23.458990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:23.625957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ecba19-0eac-4b5b-9aa4-00a25d96a90d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:23.662059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eaf8f97-785e-4bc8-b9ca-1d2fe0201ea3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:23.662959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaf8f97-785e-4bc8-b9ca-1d2fe0201ea3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:23.663243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaf8f97-785e-4bc8-b9ca-1d2fe0201ea3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:23.663524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eaf8f97-785e-4bc8-b9ca-1d2fe0201ea3 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:23.664730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaf8f97-785e-4bc8-b9ca-1d2fe0201ea3 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:23.666178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eaf8f97-785e-4bc8-b9ca-1d2fe0201ea3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:23.668226 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 147/293] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:23 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:23.675991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:23.815716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e941ac23-3292-47a9-bf78-4f50cfec68a0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:23.864322 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e52f1345-ddab-4318-b4e5-8be16a853472 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:23.865111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52f1345-ddab-4318-b4e5-8be16a853472 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:23.865394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52f1345-ddab-4318-b4e5-8be16a853472 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:23.865671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e52f1345-ddab-4318-b4e5-8be16a853472 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:23.866507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52f1345-ddab-4318-b4e5-8be16a853472 None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=112eb9a1b66347948b127628845aff42) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:23.866841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52f1345-ddab-4318-b4e5-8be16a853472 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:23.868282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e52f1345-ddab-4318-b4e5-8be16a853472 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:23.876878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 147/294] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:23 2022] GET /identity/v3/role_assignments?user.id=112eb9a1b66347948b127628845aff42&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 608 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:24.011473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add323bd-c2b2-42d8-a127-152623fe620c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.011473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add323bd-c2b2-42d8-a127-152623fe620c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.013648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-add323bd-c2b2-42d8-a127-152623fe620c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.014078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 148/295] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:24.020517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cd1f3-d8de-4c0c-8f62-9c664c1603d0 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.021120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cd1f3-d8de-4c0c-8f62-9c664c1603d0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.021625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69cd1f3-d8de-4c0c-8f62-9c664c1603d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.040971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-a69cd1f3-d8de-4c0c-8f62-9c664c1603d0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:24.042582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a69cd1f3-d8de-4c0c-8f62-9c664c1603d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:24.090082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:24.134105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 148/296] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:24.157424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add323bd-c2b2-42d8-a127-152623fe620c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.217720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:24.265842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-add323bd-c2b2-42d8-a127-152623fe620c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:24.302644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48abb228-9212-4e17-aee9-0bc50f76efb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:24.303477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48abb228-9212-4e17-aee9-0bc50f76efb6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.303764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48abb228-9212-4e17-aee9-0bc50f76efb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.304043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48abb228-9212-4e17-aee9-0bc50f76efb6 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.314793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48abb228-9212-4e17-aee9-0bc50f76efb6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-48abb228-9212-4e17-aee9-0bc50f76efb6 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:24.323175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:24.324058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:24.324855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 149/297] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:24.331031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cd1f3-d8de-4c0c-8f62-9c664c1603d0 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.332286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b3e6d-8c3f-4975-9cdd-143bdd6c373f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.332607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b3e6d-8c3f-4975-9cdd-143bdd6c373f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.332890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991b3e6d-8c3f-4975-9cdd-143bdd6c373f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.334391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 150/298] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:24.341021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c06e4f2-858d-4de0-b8c6-0cbaa813b62c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.341308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c06e4f2-858d-4de0-b8c6-0cbaa813b62c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.341587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c06e4f2-858d-4de0-b8c6-0cbaa813b62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.363473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-5c06e4f2-858d-4de0-b8c6-0cbaa813b62c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:24.364670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5c06e4f2-858d-4de0-b8c6-0cbaa813b62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:24.450894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 151/299] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:24.458156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d691db-b25f-4457-831a-5a6c9542b21b None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.458457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d691db-b25f-4457-831a-5a6c9542b21b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.458711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d691db-b25f-4457-831a-5a6c9542b21b None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.460166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 152/300] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:24.461291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69cd1f3-d8de-4c0c-8f62-9c664c1603d0 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:24.465283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8473399d-6c53-4e84-a169-ef937131036f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.465566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8473399d-6c53-4e84-a169-ef937131036f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.465842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8473399d-6c53-4e84-a169-ef937131036f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.483682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8473399d-6c53-4e84-a169-ef937131036f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:24.484866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8473399d-6c53-4e84-a169-ef937131036f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:24.500067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:24.500791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.501046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.501323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.502066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:24.502374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:24.505231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:24.514220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 149/301] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:24 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:24.519265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.533027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:24.553673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:24.562182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 153/302] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:24.563241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8473399d-6c53-4e84-a169-ef937131036f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.640797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4446fbee-5b8d-48a6-9354-1f8b1ea00a6c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:24.666633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8473399d-6c53-4e84-a169-ef937131036f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:24.676981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:24.678413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.678877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.679295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.680761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:24.682339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:24.684483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 150/303] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:24 2022] GET /identity/v3/domains/default => generated 199 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:24.685389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.704773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:24.705618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.705904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.706181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] PATH_INFO: `/v3/users/c41a13a77bd545818553456ae375acf7` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.707922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] RBAC: Authorizing `identity:get_user(user_id=c41a13a77bd545818553456ae375acf7)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:24.709500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:24.713351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 154/304] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:24 2022] GET /identity/v3/users/c41a13a77bd545818553456ae375acf7 => generated 293 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:24.713351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.767500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:24.792987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:24.797357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e0815c3-678a-4815-8496-3daef53a9656 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:24.828197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb8a58e8-0584-40e5-8b32-5bd2224fee92 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:24.831297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854818df-7190-4655-a5af-e872c047e841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:24.832197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854818df-7190-4655-a5af-e872c047e841 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.832551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854818df-7190-4655-a5af-e872c047e841 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.832904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-854818df-7190-4655-a5af-e872c047e841 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.843223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-854818df-7190-4655-a5af-e872c047e841 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-854818df-7190-4655-a5af-e872c047e841 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:24.851873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:24.852763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:24.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 151/305] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:24.854274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854818df-7190-4655-a5af-e872c047e841 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.869928 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:24.870593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:24.870909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:24.871183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] PATH_INFO: `/v3/groups` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:24.872039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:24.873475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:24.940089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 155/306] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Jul 14 11:59:24 2022] POST /identity/v3/groups => generated 229 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:24.941339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:24.976204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-854818df-7190-4655-a5af-e872c047e841 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:25.018631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:25.019717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.020003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.020286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] PATH_INFO: `/v3/system/users/c41a13a77bd545818553456ae375acf7/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.022751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=c41a13a77bd545818553456ae375acf7, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:25.024484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:25.084091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf1b7fec-4610-4ac3-9149-28253f019764 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:25.086416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:25.094738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 152/307] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Jul 14 11:59:24 2022] PUT /identity/v3/system/users/c41a13a77bd545818553456ae375acf7/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:25.141547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5d1d50-006f-40bf-95bf-b48c1573725b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:25.142297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d1d50-006f-40bf-95bf-b48c1573725b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.142580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d1d50-006f-40bf-95bf-b48c1573725b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.142946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5d1d50-006f-40bf-95bf-b48c1573725b None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.144500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5d1d50-006f-40bf-95bf-b48c1573725b None admin] RBAC: Adding query filter params (name=reader) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:25.144500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5d1d50-006f-40bf-95bf-b48c1573725b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:25.146382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5d1d50-006f-40bf-95bf-b48c1573725b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:25.156725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 156/308] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:25.163123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:25.195527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:25.286523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07d28453-23ab-4ee4-a618-454a66aee2c4 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:25.325416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:25.326295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.326580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.326879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] PATH_INFO: `/v3/users/a8d59a1ec9e1475488c82deb0a5b4c6e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.340027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142aefeb-f7da-4a34-979b-f2977ee77b5d None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.340341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142aefeb-f7da-4a34-979b-f2977ee77b5d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.340597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142aefeb-f7da-4a34-979b-f2977ee77b5d None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.341580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] RBAC: Authorizing `identity:get_user(user_id=a8d59a1ec9e1475488c82deb0a5b4c6e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:25.342251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 157/309] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:25.343155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:25.345185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 153/310] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:25 2022] GET /identity/v3/users/a8d59a1ec9e1475488c82deb0a5b4c6e => generated 293 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:25.347334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122f10f5-2ae2-47ec-83ae-8b877f9382ad None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.347689 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122f10f5-2ae2-47ec-83ae-8b877f9382ad None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.348016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122f10f5-2ae2-47ec-83ae-8b877f9382ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.351105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:25.366682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-122f10f5-2ae2-47ec-83ae-8b877f9382ad None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:25.367949 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-122f10f5-2ae2-47ec-83ae-8b877f9382ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:25.453526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 158/311] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:25.457532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:25.470135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a218fe-02d8-407b-a18c-52010cd80ddb None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:25.477565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122f10f5-2ae2-47ec-83ae-8b877f9382ad None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:25.514030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:25.515633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.515931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.516216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.517067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] RBAC: Adding query filter params (role.id=ae7f6ead5e04485fa45d5b011096569d, user.id=a8d59a1ec9e1475488c82deb0a5b4c6e, scope.system=all) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:25.517385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:25.518852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:25.526870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 154/312] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Jul 14 11:59:25 2022] GET /identity/v3/role_assignments?user.id=a8d59a1ec9e1475488c82deb0a5b4c6e&scope.system=all&role.id=ae7f6ead5e04485fa45d5b011096569d => generated 226 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:25.595463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:25.599011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122f10f5-2ae2-47ec-83ae-8b877f9382ad None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:25.639263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:25.640081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.640359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.640632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.652896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:25.661685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:25.662616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:25.663471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 159/313] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:25.669019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:25.794032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe96397-1aea-4a67-9270-967e9e8f00c3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:25.829882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:25.830553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.830858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.831136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.831889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:25.832510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:25.834013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:25.846952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 155/314] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:25.851848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:25.956566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da7613e5-8a1c-4e9f-acee-03d88ebc2465 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:25.991894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:25.992765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:25.993046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:25.993316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] PATH_INFO: `/v3/users/fe13d9b013b342cab05d11a5bc81fd45` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:25.994528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] RBAC: Authorizing `identity:get_user(user_id=fe13d9b013b342cab05d11a5bc81fd45)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:25.996029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:25.997932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 160/315] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:25 2022] GET /identity/v3/users/fe13d9b013b342cab05d11a5bc81fd45 => generated 297 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:26.003246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:26.013535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:26.111028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f149c5-1b3a-4cbd-8f39-aea220e1e0c0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:26.148621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:26.149878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:26.150355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:26.150851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:26.224959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:26.226743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:26.235134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:26.335911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 156/316] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:26.346792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:26.371485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:26.446644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ae1a8a2-3f1c-46b4-b899-9f0d92004aa1 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:26.480733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc9f667-212e-4e97-a758-5442b2d1e01e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:26.481600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc9f667-212e-4e97-a758-5442b2d1e01e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:26.481881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc9f667-212e-4e97-a758-5442b2d1e01e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:26.482157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc9f667-212e-4e97-a758-5442b2d1e01e None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:26.483451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc9f667-212e-4e97-a758-5442b2d1e01e None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:26.484886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc9f667-212e-4e97-a758-5442b2d1e01e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:26.486957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 161/317] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:26 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:26.492645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:26.594211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd4b132-c0c0-4799-9a5c-f0cc54d4d22d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:26.604671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:26.631046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e621aaf-c298-4735-9c4a-f91e10f67075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:26.631677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e621aaf-c298-4735-9c4a-f91e10f67075 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:26.631959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e621aaf-c298-4735-9c4a-f91e10f67075 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:26.632266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e621aaf-c298-4735-9c4a-f91e10f67075 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:26.633090 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e621aaf-c298-4735-9c4a-f91e10f67075 None admin] RBAC: Adding query filter params (role.id=ae7f6ead5e04485fa45d5b011096569d, scope.project.id=5586dcebfe8c4e81bb952f00a518722e, user.id=fe13d9b013b342cab05d11a5bc81fd45) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:26.633402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e621aaf-c298-4735-9c4a-f91e10f67075 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:26.634928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e621aaf-c298-4735-9c4a-f91e10f67075 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:26.641233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 157/318] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:26 2022] GET /identity/v3/role_assignments?user.id=fe13d9b013b342cab05d11a5bc81fd45&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=ae7f6ead5e04485fa45d5b011096569d => generated 608 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:27.151479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28a860-c226-4f0e-a36e-6bb6d7b733ca None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.151754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28a860-c226-4f0e-a36e-6bb6d7b733ca None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.152143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae28a860-c226-4f0e-a36e-6bb6d7b733ca None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.153697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 162/319] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:27.158545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16795dd-b76d-4717-9f8e-188bc2190a4c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.159009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16795dd-b76d-4717-9f8e-188bc2190a4c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.159169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16795dd-b76d-4717-9f8e-188bc2190a4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.177063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b16795dd-b76d-4717-9f8e-188bc2190a4c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:27.178261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b16795dd-b76d-4717-9f8e-188bc2190a4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:27.204543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0532a31-39ce-4072-bc8f-af179531265d None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.204826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0532a31-39ce-4072-bc8f-af179531265d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.205101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0532a31-39ce-4072-bc8f-af179531265d None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.206470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 163/320] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:59:27.211391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa815517-28ee-4e1d-ba60-23bf2956d07b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.211681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa815517-28ee-4e1d-ba60-23bf2956d07b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.211961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa815517-28ee-4e1d-ba60-23bf2956d07b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.228615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-aa815517-28ee-4e1d-ba60-23bf2956d07b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:27.229812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-aa815517-28ee-4e1d-ba60-23bf2956d07b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:27.251523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 158/321] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:27.274717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16795dd-b76d-4717-9f8e-188bc2190a4c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:27.299528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 164/322] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:27.323018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa815517-28ee-4e1d-ba60-23bf2956d07b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:27.344840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:27.380087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:27.383367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16795dd-b76d-4717-9f8e-188bc2190a4c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:27.420510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:27.421213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.421498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.421778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.422988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:27.424441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:27.426174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 159/323] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:27.431551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:27.433232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa815517-28ee-4e1d-ba60-23bf2956d07b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:27.470655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:27.471829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.471829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.472097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.482849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:27.491297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:27.492167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:27.492942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 165/324] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:27.498204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:27.539403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a73ab8ed-f3d3-4b3a-882d-d8b82db55246 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:27.574000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825dd62b-06bb-4379-9e1d-a876bde794ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:27.574670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825dd62b-06bb-4379-9e1d-a876bde794ad None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.574998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825dd62b-06bb-4379-9e1d-a876bde794ad None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.575275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825dd62b-06bb-4379-9e1d-a876bde794ad None admin] PATH_INFO: `/v3/groups` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.575968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825dd62b-06bb-4379-9e1d-a876bde794ad None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:27.577414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825dd62b-06bb-4379-9e1d-a876bde794ad None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:27.624534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:27.643682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d9d28b-c0b3-4415-bd45-e916315979d8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:27.653202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 160/325] 10.222.0.27 () {60 vars in 1134 bytes} [Thu Jul 14 11:59:27 2022] POST /identity/v3/groups => generated 226 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 11:59:27.653202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f460b05c-1384-4d92-b556-50eb4db96953 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.653202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f460b05c-1384-4d92-b556-50eb4db96953 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.653202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f460b05c-1384-4d92-b556-50eb4db96953 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.653202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 161/326] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:27.653202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60073bb7-92f0-4cdd-a6b5-048dc1ca4e85 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.653202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60073bb7-92f0-4cdd-a6b5-048dc1ca4e85 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.663458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60073bb7-92f0-4cdd-a6b5-048dc1ca4e85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.681674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:27.682083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-60073bb7-92f0-4cdd-a6b5-048dc1ca4e85 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:27.682183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.682183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.682419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.682517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-60073bb7-92f0-4cdd-a6b5-048dc1ca4e85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:27.683221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:27.683535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:27.684959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:27.695180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 166/327] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:27.700866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:27.716778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:27.766254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 162/328] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:27.789786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60073bb7-92f0-4cdd-a6b5-048dc1ca4e85 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:27.831281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693bf4f9-de59-4b74-aebd-262df48ea937 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:27.870682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:27.871607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.871894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.872170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] PATH_INFO: `/v3/users/c41a13a77bd545818553456ae375acf7` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.873481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] RBAC: Authorizing `identity:get_user(user_id=c41a13a77bd545818553456ae375acf7)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:27.874999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:27.877788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 167/329] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:27 2022] GET /identity/v3/users/c41a13a77bd545818553456ae375acf7 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:27.883791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:27.897344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:27.917691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60073bb7-92f0-4cdd-a6b5-048dc1ca4e85 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:27.965856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:27.966667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:27.966996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:27.967273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:27.980000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:27.987901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:27.989028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:27.990007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 163/330] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:27 2022] GET /identity/v3/roles/reader => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:27.995103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:28.009615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210934e5-4e74-4b7c-be02-86cadb3e943b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:28.016352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:28.049404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:28.050233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:28.050746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:28.051191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:28.052176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] RBAC: Adding query filter params (role.id=56f2092f7f5e42b09679dec166d174c5, user.id=c41a13a77bd545818553456ae375acf7, scope.system=all) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:28.052569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:28.054209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:28.070747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 168/331] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Jul 14 11:59:27 2022] GET /identity/v3/role_assignments?user.id=c41a13a77bd545818553456ae375acf7&scope.system=all&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 510 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:28.127316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d950f7-304d-4e20-9006-cecfb6b90dc6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:28.167497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:28.168961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:28.169428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:28.169886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:28.170973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:28.171515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:28.173164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:28.183534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 164/332] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:27 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:28.189360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:28.272889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:28.309316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31ea6ed-8486-4565-9039-5614b89cda4b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:28.352317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fba946-3a55-4068-a9c9-c61c863977cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:28.353205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fba946-3a55-4068-a9c9-c61c863977cc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:28.353487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fba946-3a55-4068-a9c9-c61c863977cc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:28.353764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fba946-3a55-4068-a9c9-c61c863977cc None admin] PATH_INFO: `/v3/users/a8d59a1ec9e1475488c82deb0a5b4c6e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:28.355010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fba946-3a55-4068-a9c9-c61c863977cc None admin] RBAC: Authorizing `identity:get_user(user_id=a8d59a1ec9e1475488c82deb0a5b4c6e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:28.356454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fba946-3a55-4068-a9c9-c61c863977cc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:28.358099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 169/333] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:28 2022] GET /identity/v3/users/a8d59a1ec9e1475488c82deb0a5b4c6e => generated 293 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:28.364495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:28.455265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:28.477021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-268b70cb-dcef-4b38-a8d4-5ae133fc09d1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:28.514218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2205ae31-b4fd-4c14-af5c-63672a1181fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:28.515119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2205ae31-b4fd-4c14-af5c-63672a1181fd None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:28.515406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2205ae31-b4fd-4c14-af5c-63672a1181fd None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:28.515687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2205ae31-b4fd-4c14-af5c-63672a1181fd None admin] PATH_INFO: `/v3/system/users/a8d59a1ec9e1475488c82deb0a5b4c6e/roles/ae7f6ead5e04485fa45d5b011096569d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:28.517215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2205ae31-b4fd-4c14-af5c-63672a1181fd None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=a8d59a1ec9e1475488c82deb0a5b4c6e, role_id=ae7f6ead5e04485fa45d5b011096569d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:28.518618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2205ae31-b4fd-4c14-af5c-63672a1181fd None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:28.609221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 165/334] 10.222.0.27 () {58 vars in 1334 bytes} [Thu Jul 14 11:59:28 2022] PUT /identity/v3/system/users/a8d59a1ec9e1475488c82deb0a5b4c6e/roles/ae7f6ead5e04485fa45d5b011096569d => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:30.286013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bba55b-1e06-4967-b255-c37908d4cb5a None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:30.337479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bba55b-1e06-4967-b255-c37908d4cb5a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:30.337479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bba55b-1e06-4967-b255-c37908d4cb5a None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:30.337479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 170/335] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:30.337756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3952e360-1d80-442c-82eb-595d67c0af63 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:30.337756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3952e360-1d80-442c-82eb-595d67c0af63 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:30.337756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3952e360-1d80-442c-82eb-595d67c0af63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:30.337756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3952e360-1d80-442c-82eb-595d67c0af63 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:30.337756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3952e360-1d80-442c-82eb-595d67c0af63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:30.412981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 166/336] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:30.455626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bba55b-1e06-4967-b255-c37908d4cb5a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:30.589421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bba55b-1e06-4967-b255-c37908d4cb5a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:30.629018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:30.629824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:30.630105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:30.630379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:30.637265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:30.649256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:30.657068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:30.657977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:30.658839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 171/337] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:30 2022] GET /identity/v3/roles/member => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:30.665136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3952e360-1d80-442c-82eb-595d67c0af63 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:30.762543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:30.797579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3952e360-1d80-442c-82eb-595d67c0af63 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:30.843361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6777a92-39c1-4c60-8349-0afcb7016b58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:30.843959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6777a92-39c1-4c60-8349-0afcb7016b58 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:30.844243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6777a92-39c1-4c60-8349-0afcb7016b58 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:30.844523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6777a92-39c1-4c60-8349-0afcb7016b58 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:30.845285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6777a92-39c1-4c60-8349-0afcb7016b58 None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:30.845598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6777a92-39c1-4c60-8349-0afcb7016b58 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:30.847174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6777a92-39c1-4c60-8349-0afcb7016b58 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:30.857186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 167/338] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:30 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:30.863613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:30.993784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46979697-4cea-47fb-88a6-9330e9944314 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:31.034081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:31.034727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.036002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.036002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.116408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:31.123678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:31.126915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:31.172874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bea192-19b1-40df-85b0-184c9282db21 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.173521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bea192-19b1-40df-85b0-184c9282db21 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.174024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59bea192-19b1-40df-85b0-184c9282db21 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.175993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 168/339] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:59:31.181861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819d578-00e2-4a7c-bebb-76fcf3c569ff None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.182433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819d578-00e2-4a7c-bebb-76fcf3c569ff None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.183054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a819d578-00e2-4a7c-bebb-76fcf3c569ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.211325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-a819d578-00e2-4a7c-bebb-76fcf3c569ff None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:31.212966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a819d578-00e2-4a7c-bebb-76fcf3c569ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:31.239948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 172/340] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:30 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:31.247953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:31.308499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 169/341] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:31.350633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819d578-00e2-4a7c-bebb-76fcf3c569ff None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:31.402281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:31.411855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d186b30-2269-42a3-979c-4ac1a9bf0944 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:31.439012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:31.455411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ce0eaf-ed41-4829-8bfb-2a0112130d20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:31.456688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ce0eaf-ed41-4829-8bfb-2a0112130d20 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.457550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ce0eaf-ed41-4829-8bfb-2a0112130d20 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.457942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ce0eaf-ed41-4829-8bfb-2a0112130d20 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.459671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ce0eaf-ed41-4829-8bfb-2a0112130d20 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:31.461990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ce0eaf-ed41-4829-8bfb-2a0112130d20 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:31.464650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 173/342] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:31 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:31.465967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a42296-6481-40ac-929f-00a603e4e163 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.466327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a42296-6481-40ac-929f-00a603e4e163 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.466680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8a42296-6481-40ac-929f-00a603e4e163 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.472389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 174/343] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:31.472389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1175fd34-9df3-4916-b2fa-5f4ab6b6e081 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.472389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1175fd34-9df3-4916-b2fa-5f4ab6b6e081 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.472389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1175fd34-9df3-4916-b2fa-5f4ab6b6e081 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.473354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 175/344] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:31.474437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1175fd34-9df3-4916-b2fa-5f4ab6b6e081 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:31.487063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a819d578-00e2-4a7c-bebb-76fcf3c569ff None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:31.538522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f51762c-e053-4d18-a72d-7e9500226020 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:31.539375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f51762c-e053-4d18-a72d-7e9500226020 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.539655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f51762c-e053-4d18-a72d-7e9500226020 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.539920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f51762c-e053-4d18-a72d-7e9500226020 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.555111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f51762c-e053-4d18-a72d-7e9500226020 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-7f51762c-e053-4d18-a72d-7e9500226020 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:31.567044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:31.567965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:31.568736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 170/345] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:31.569931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072e240-4338-4bd1-9488-6936dd841752 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.569931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072e240-4338-4bd1-9488-6936dd841752 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.570088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4072e240-4338-4bd1-9488-6936dd841752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.596572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4072e240-4338-4bd1-9488-6936dd841752 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:31.598657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4072e240-4338-4bd1-9488-6936dd841752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:31.609089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1175fd34-9df3-4916-b2fa-5f4ab6b6e081 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:31.650606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123854d0-6a78-409f-825f-f4a5c066c788 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:31.651554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123854d0-6a78-409f-825f-f4a5c066c788 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.651837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123854d0-6a78-409f-825f-f4a5c066c788 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.652168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123854d0-6a78-409f-825f-f4a5c066c788 None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.658880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123854d0-6a78-409f-825f-f4a5c066c788 None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:31.661038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123854d0-6a78-409f-825f-f4a5c066c788 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:31.662850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 176/346] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:31 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:31.664054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a18399-e327-41e5-9a27-d6025e600ba2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.664440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a18399-e327-41e5-9a27-d6025e600ba2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.664735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60a18399-e327-41e5-9a27-d6025e600ba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.686172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-60a18399-e327-41e5-9a27-d6025e600ba2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:31.688373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-60a18399-e327-41e5-9a27-d6025e600ba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:31.700037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 171/347] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:31.700973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4072e240-4338-4bd1-9488-6936dd841752 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:31.721312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:31.779161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 177/348] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:31.781663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a18399-e327-41e5-9a27-d6025e600ba2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:31.804136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:31.863387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4072e240-4338-4bd1-9488-6936dd841752 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:31.905339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e1e413-d33b-471e-b57e-c4d6210a99de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:31.906017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e1e413-d33b-471e-b57e-c4d6210a99de None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.906299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e1e413-d33b-471e-b57e-c4d6210a99de None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.906574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e1e413-d33b-471e-b57e-c4d6210a99de None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.907066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60a18399-e327-41e5-9a27-d6025e600ba2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:31.907385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e1e413-d33b-471e-b57e-c4d6210a99de None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:31.907694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e1e413-d33b-471e-b57e-c4d6210a99de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:31.909546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e1e413-d33b-471e-b57e-c4d6210a99de None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:31.922845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 172/349] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:31.924112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8470dca9-fb55-48e7-bf61-f41daf5d156e None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.924398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8470dca9-fb55-48e7-bf61-f41daf5d156e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.924673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8470dca9-fb55-48e7-bf61-f41daf5d156e None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.926548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 173/350] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:31.930442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8470dca9-fb55-48e7-bf61-f41daf5d156e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:31.953633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:31.954295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:31.954576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:31.954881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:31.955724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:31.956037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:31.958151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:31.971266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 178/351] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:31 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 260 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:31.972148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:32.009733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:32.074981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8470dca9-fb55-48e7-bf61-f41daf5d156e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:32.094890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:32.113780 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:32.115076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.115550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ca002d-c6b8-45d4-a752-dd9ad2b293ff None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:32.115777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.116242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.129329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:32.137621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:32.138585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:32.139423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:32.141900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 174/352] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:32.142989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:32.157280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443b4822-f13f-4601-a20e-a404ff3bffd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:32.158179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b4822-f13f-4601-a20e-a404ff3bffd6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.158649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b4822-f13f-4601-a20e-a404ff3bffd6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.158752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b4822-f13f-4601-a20e-a404ff3bffd6 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.171097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443b4822-f13f-4601-a20e-a404ff3bffd6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-443b4822-f13f-4601-a20e-a404ff3bffd6 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:32.179284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:32.180182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:32.180987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 179/353] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:32.181730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdcbb44-4368-4325-8b98-3e04f483ed3e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.182051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdcbb44-4368-4325-8b98-3e04f483ed3e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.182255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcdcbb44-4368-4325-8b98-3e04f483ed3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.199589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-fcdcbb44-4368-4325-8b98-3e04f483ed3e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:32.200998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fcdcbb44-4368-4325-8b98-3e04f483ed3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:32.273127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a6dada-5a27-4bb5-9bd6-86b367549025 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:32.278912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 180/354] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:32.278912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdcbb44-4368-4325-8b98-3e04f483ed3e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:32.291715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:32.331717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:32.332926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.333210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.333482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.385545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcdcbb44-4368-4325-8b98-3e04f483ed3e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:32.389024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:32.413077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:32.414553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:32.424396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:32.425885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.426374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.426906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.427878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:32.428412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:32.430296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:32.440097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 181/355] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:32 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:32.442915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:32.537652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:32.543516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 175/356] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:32.544889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:32.570711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0554a50c-5071-467b-ab86-54358b082688 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:32.642207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d032a26e-36e1-4363-a856-5b27d80c5b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:32.642991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032a26e-36e1-4363-a856-5b27d80c5b23 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.643275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032a26e-36e1-4363-a856-5b27d80c5b23 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.643988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d032a26e-36e1-4363-a856-5b27d80c5b23 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.644763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d032a26e-36e1-4363-a856-5b27d80c5b23 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:32.645075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d032a26e-36e1-4363-a856-5b27d80c5b23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:32.646838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d032a26e-36e1-4363-a856-5b27d80c5b23 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:32.657859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 182/357] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:32.659120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d624756-c27a-4865-85d7-b9a60f997b19 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.659409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d624756-c27a-4865-85d7-b9a60f997b19 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.659687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d624756-c27a-4865-85d7-b9a60f997b19 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.662336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 183/358] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:32.663017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d624756-c27a-4865-85d7-b9a60f997b19 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:32.703942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:32.740541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7afb60c3-0057-48b2-a385-d8640400cdd9 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:32.789318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:32.790361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.790809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.791233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.796676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d624756-c27a-4865-85d7-b9a60f997b19 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:32.806217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:32.816948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:32.817936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:32.818750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 176/359] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:32 2022] GET /identity/v3/roles/reader => generated 84 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:32.819582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:32.840594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8d6479-10d9-4bc0-bc9a-1361a5ca848e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:32.841266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d6479-10d9-4bc0-bc9a-1361a5ca848e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:32.841606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d6479-10d9-4bc0-bc9a-1361a5ca848e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:32.841898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8d6479-10d9-4bc0-bc9a-1361a5ca848e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:32.846641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:32.936556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8d6479-10d9-4bc0-bc9a-1361a5ca848e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:32.938530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8d6479-10d9-4bc0-bc9a-1361a5ca848e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:32.963128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31b884ae-5dc8-41f8-a8b8-e94ce24eb730 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.009464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.010479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.010794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.011076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.012796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:33.014255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:33.016234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 177/360] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:32 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:33.016978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.027389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:33.079745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 184/361] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:33.081426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18acb410-a57c-452b-8789-a107b0a6cdd0 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.082812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18acb410-a57c-452b-8789-a107b0a6cdd0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.086313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18acb410-a57c-452b-8789-a107b0a6cdd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.107115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-18acb410-a57c-452b-8789-a107b0a6cdd0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:33.108428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-18acb410-a57c-452b-8789-a107b0a6cdd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:33.144700 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97a9fa2-be7f-4698-b91c-e38f19e9fc0e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.149258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:33.192802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.193472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.193755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.194029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.200821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 185/362] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:33.201509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18acb410-a57c-452b-8789-a107b0a6cdd0 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.283890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:33.285508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:33.293846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:33.345519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18acb410-a57c-452b-8789-a107b0a6cdd0 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.392241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.394832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.394832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.394832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.395117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:33.395212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:33.396577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:33.410842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 186/363] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:33.411882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.420478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 178/364] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:33.430264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.452345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:33.552353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0d118f-cccd-4cf0-9f76-5c87aa744fe8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.556233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b2e2c0-5425-4f8f-b65c-fd5e482b9926 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.564022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:33.596580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.599154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.599756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.600184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.601629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:33.603290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:33.605229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.605379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 179/365] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:33.606179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.606577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.606947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.608074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.608227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:33.609847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:33.612128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 187/366] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:33.613985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.734681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:33.740285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05150f6-4c69-4e61-ab93-0e044fbf8960 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.746952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4991015c-4ff0-4dc9-a821-e5ace4ce4ba2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.782974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.784016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.784300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.784936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] PATH_INFO: `/v3/users/a8d59a1ec9e1475488c82deb0a5b4c6e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.786416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] RBAC: Authorizing `identity:get_user(user_id=a8d59a1ec9e1475488c82deb0a5b4c6e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:33.787922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:33.790395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 188/367] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/users/a8d59a1ec9e1475488c82deb0a5b4c6e => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:33.790395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.796175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.798302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.798789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.799249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.820811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:33.828606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:33.829701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 180/368] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:33.830656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:33.874660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:33.922275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5248130-6b77-4ba2-a2d8-991a6b266a0c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.967857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed8412e9-2689-4b6a-9580-6c357564cacb None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:33.972872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:33.972872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:33.972872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:33.972872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:33.974206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:33.976128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:33.977781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 189/369] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:33.978450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:34.011236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:34.019251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.020126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.024811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.024811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.024811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.024811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.026485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 181/370] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:34.027141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:34.105762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc4930b5-d5f8-4e04-b172-dd511e26b589 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:34.145861 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.146511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.146808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.147087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.147914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:34.148220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.149904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.155263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f93813-3950-437b-a6e9-7ce8af9e97c2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:34.162430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 190/371] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:33 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=1ff35caa6d284df39cd287349f66c567 => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:34.162430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:34.163711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:34.200805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.201455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.201741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.202017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.202907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] RBAC: Adding query filter params (role.id=ae7f6ead5e04485fa45d5b011096569d, user.id=a8d59a1ec9e1475488c82deb0a5b4c6e, scope.system=all) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:34.203226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.204730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.219482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 182/372] 10.222.0.27 () {56 vars in 1308 bytes} [Thu Jul 14 11:59:34 2022] GET /identity/v3/role_assignments?user.id=a8d59a1ec9e1475488c82deb0a5b4c6e&scope.system=all&role.id=ae7f6ead5e04485fa45d5b011096569d => generated 510 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:34.220424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:34.290630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:34.300156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a557433d-ddfc-452b-87df-87a6447d03a0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:34.357550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.358153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.358440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.358720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.359651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:34.360082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.361595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.369878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4e875b9-8420-4294-a29d-0f9f4d1d4826 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:34.376490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 191/373] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:34 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:34.376490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:34.418535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee72cc-5b82-45aa-96a7-b0c0539be5fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.419725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee72cc-5b82-45aa-96a7-b0c0539be5fa None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.420148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee72cc-5b82-45aa-96a7-b0c0539be5fa None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.420578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee72cc-5b82-45aa-96a7-b0c0539be5fa None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.421847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee72cc-5b82-45aa-96a7-b0c0539be5fa None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.423798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee72cc-5b82-45aa-96a7-b0c0539be5fa None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.430138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 183/374] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:34 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:34.430138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e32e65-2fbd-44aa-97ac-3bf8bd44ae3a None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.430138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e32e65-2fbd-44aa-97ac-3bf8bd44ae3a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.430138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e32e65-2fbd-44aa-97ac-3bf8bd44ae3a None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.430811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 184/375] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:34.432518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e32e65-2fbd-44aa-97ac-3bf8bd44ae3a None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:34.466353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:34.510555 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab9d27c-7b42-4384-986f-9a33e44bda1a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:34.560756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471537ab-1421-4018-8691-e434e1360511 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.561721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471537ab-1421-4018-8691-e434e1360511 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.562147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471537ab-1421-4018-8691-e434e1360511 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.562431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471537ab-1421-4018-8691-e434e1360511 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.563300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471537ab-1421-4018-8691-e434e1360511 None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=5586dcebfe8c4e81bb952f00a518722e) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:34.563611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471537ab-1421-4018-8691-e434e1360511 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.565152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471537ab-1421-4018-8691-e434e1360511 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.565714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e32e65-2fbd-44aa-97ac-3bf8bd44ae3a None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:34.572082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 192/376] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:34 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 260 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:34.572900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471537ab-1421-4018-8691-e434e1360511 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:34.607078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:34.621518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.621518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.621518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.621518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.742512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.744465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.745666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471537ab-1421-4018-8691-e434e1360511 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:34.790994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f58f1cd-f4c5-4c3c-8800-5242c807c3e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:34.792032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58f1cd-f4c5-4c3c-8800-5242c807c3e8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.792318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58f1cd-f4c5-4c3c-8800-5242c807c3e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.792597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f58f1cd-f4c5-4c3c-8800-5242c807c3e8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.793435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f58f1cd-f4c5-4c3c-8800-5242c807c3e8 None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=5586dcebfe8c4e81bb952f00a518722e) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:34.793750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f58f1cd-f4c5-4c3c-8800-5242c807c3e8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:34.795232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f58f1cd-f4c5-4c3c-8800-5242c807c3e8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:34.796230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:34.803350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 193/377] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:34 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=1ff35caa6d284df39cd287349f66c567 => generated 260 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:34.809982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd84128-d0da-4619-9562-11e6de7b48ed None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:34.809982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd84128-d0da-4619-9562-11e6de7b48ed None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:34.809982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd84128-d0da-4619-9562-11e6de7b48ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:34.827175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-edd84128-d0da-4619-9562-11e6de7b48ed None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:34.828970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-edd84128-d0da-4619-9562-11e6de7b48ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:34.958282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 185/378] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 527 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:34.964259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 194/379] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:34.965192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:35.011136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd84128-d0da-4619-9562-11e6de7b48ed None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:35.042820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:35.140762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a1d409d-656c-4d8d-8b3a-3411a0c58e5d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:35.172290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:35.181574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:35.182879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:35.183297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:35.183871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] PATH_INFO: `/v3/projects/1074c03ef1614e229d46654abff5220e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:35.185265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] RBAC: Authorizing `identity:get_project(project_id=1074c03ef1614e229d46654abff5220e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:35.187006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:35.189437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 186/380] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:34 2022] GET /identity/v3/projects/1074c03ef1614e229d46654abff5220e => generated 344 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:35.197128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd84128-d0da-4619-9562-11e6de7b48ed None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:35.197277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:35.260923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:35.261651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:35.261937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:35.262216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:35.295681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:35.312163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:35.313358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:35.314362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 195/381] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:35.320039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce372355-3db8-4798-a895-62f198c82920 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:35.328775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:35.388541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:35.389400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:35.389681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:35.389956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] PATH_INFO: `/v3/groups/13ab8d5d3af749f8b651e1af924066d9` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:35.399063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] RBAC: Authorizing `identity:get_group(group_id=13ab8d5d3af749f8b651e1af924066d9)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:35.399329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:35.401313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 187/382] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:35 2022] GET /identity/v3/groups/13ab8d5d3af749f8b651e1af924066d9 => generated 229 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:35.411719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:35.459111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:35.504484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-195460b6-ec5f-43ad-91de-e0039ddddf40 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:35.519788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:35.554567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:35.555281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:35.555540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:35.555872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:35.556617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:35.556921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:35.558289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:35.567458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 196/383] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:35.573679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:35.610525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e047a0-5a2b-479a-aa08-1c8306422131 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:35.687475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:35.688501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:35.689018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:35.689509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:35.690552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] RBAC: Adding query filter params (group.id=13ab8d5d3af749f8b651e1af924066d9, role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=1074c03ef1614e229d46654abff5220e) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:35.695234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:35.696986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:35.712963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 188/384] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:35 2022] GET /identity/v3/role_assignments?group.id=13ab8d5d3af749f8b651e1af924066d9&scope.project.id=1074c03ef1614e229d46654abff5220e&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 260 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:35.725235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8d59b9-0608-438c-8b11-bed32ff57806 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:35.770068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:35.770892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:35.771170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:35.771446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:35.772737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:35.774480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:35.776516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 197/385] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:35 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:35.781674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:35.946177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba903de-d5ec-4701-977e-f94a303fb8d0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:35.974238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:35.985135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6b1374-7e36-4ec4-ad36-770ae3d5d972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:35.995609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b1374-7e36-4ec4-ad36-770ae3d5d972 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:35.995609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b1374-7e36-4ec4-ad36-770ae3d5d972 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:35.995609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6b1374-7e36-4ec4-ad36-770ae3d5d972 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:35.996016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6b1374-7e36-4ec4-ad36-770ae3d5d972 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:35.997653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6b1374-7e36-4ec4-ad36-770ae3d5d972 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:35.999628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 189/386] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:35 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:36.005466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:36.023448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:36.145779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06773ba-478d-466a-af0e-cb6f176e5eae None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:36.181932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8348f628-40d6-4210-aa61-9c56a5ccfcfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:36.182837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348f628-40d6-4210-aa61-9c56a5ccfcfd None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:36.183119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348f628-40d6-4210-aa61-9c56a5ccfcfd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:36.183390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8348f628-40d6-4210-aa61-9c56a5ccfcfd None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/groups/3085338af3414566ac41287ecb12462e/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:36.185333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348f628-40d6-4210-aa61-9c56a5ccfcfd None admin] RBAC: Authorizing `identity:create_grant(project_id=696f929ba4b64cd88ceb4cdaa2f61d36, group_id=3085338af3414566ac41287ecb12462e, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:36.186794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8348f628-40d6-4210-aa61-9c56a5ccfcfd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:36.282509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 198/387] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Jul 14 11:59:36 2022] PUT /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/groups/3085338af3414566ac41287ecb12462e/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:36.686442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c138635-5e3f-4b06-997e-a621a593ff74 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:36.687465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c138635-5e3f-4b06-997e-a621a593ff74 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:36.687956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c138635-5e3f-4b06-997e-a621a593ff74 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:36.689535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 190/388] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:36.695760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1883579-5391-46db-a8c4-d235cc73915e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:36.696209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1883579-5391-46db-a8c4-d235cc73915e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:36.696400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1883579-5391-46db-a8c4-d235cc73915e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:36.713624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e1883579-5391-46db-a8c4-d235cc73915e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:36.718871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e1883579-5391-46db-a8c4-d235cc73915e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:36.789916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:36.797876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 199/389] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:36.838407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c138635-5e3f-4b06-997e-a621a593ff74 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:36.964463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c138635-5e3f-4b06-997e-a621a593ff74 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.011458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.011708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da9d70-f433-40f4-88ac-c4d67426d070 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.012261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da9d70-f433-40f4-88ac-c4d67426d070 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.012376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.012547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.012822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.013227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9da9d70-f433-40f4-88ac-c4d67426d070 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.018914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 200/390] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:37 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:37.021461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe64525d-4cec-4e35-8aef-f7014c614994 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.021954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe64525d-4cec-4e35-8aef-f7014c614994 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.022403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe64525d-4cec-4e35-8aef-f7014c614994 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.025188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:37.034131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:37.035141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:37.035931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:37.040301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 191/391] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:37.041159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.048217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-fe64525d-4cec-4e35-8aef-f7014c614994 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:37.049653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fe64525d-4cec-4e35-8aef-f7014c614994 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:37.107164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:37.173400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49505c6f-0f11-459b-afad-84ad2e6dd4d8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.192282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 201/392] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:37.208701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.209379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.209666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.209942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.210690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:37.211033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:37.212414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:37.214986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe64525d-4cec-4e35-8aef-f7014c614994 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.221320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 192/393] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:37.228930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.356484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe64525d-4cec-4e35-8aef-f7014c614994 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.375063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4309534-32ea-431d-8f22-1df3b7b5fe6a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.385673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:37.393592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:37.396771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01a60a5b-d518-42d2-b528-80b0a2869967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.397637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a60a5b-d518-42d2-b528-80b0a2869967 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.399470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a60a5b-d518-42d2-b528-80b0a2869967 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.399470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01a60a5b-d518-42d2-b528-80b0a2869967 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.409416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01a60a5b-d518-42d2-b528-80b0a2869967 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:37.416409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e66317-90bd-4e4a-b058-164f6f4bbb53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.417466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e66317-90bd-4e4a-b058-164f6f4bbb53 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.417895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e66317-90bd-4e4a-b058-164f6f4bbb53 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.418325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e66317-90bd-4e4a-b058-164f6f4bbb53 None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-01a60a5b-d518-42d2-b528-80b0a2869967 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:37.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:37.420356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:37.421359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:37.422500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e66317-90bd-4e4a-b058-164f6f4bbb53 None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:37.423585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 202/394] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:37.424299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e66317-90bd-4e4a-b058-164f6f4bbb53 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:37.425794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766ab52a-e745-4bd9-a327-41a1500262f9 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.426361 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766ab52a-e745-4bd9-a327-41a1500262f9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.426712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766ab52a-e745-4bd9-a327-41a1500262f9 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.432707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 193/395] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:37.432817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 203/396] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:37.434681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1998f87d-25f7-4337-b794-c645289f9321 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.434992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1998f87d-25f7-4337-b794-c645289f9321 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.435270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1998f87d-25f7-4337-b794-c645289f9321 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.435770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766ab52a-e745-4bd9-a327-41a1500262f9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.458708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1998f87d-25f7-4337-b794-c645289f9321 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:37.459969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1998f87d-25f7-4337-b794-c645289f9321 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:37.553217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766ab52a-e745-4bd9-a327-41a1500262f9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.559125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 194/397] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:37.560303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1998f87d-25f7-4337-b794-c645289f9321 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.614436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.615364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.615653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.616440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.617658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:37.619886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:37.626509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 204/398] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:37.626509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.697992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:37.698622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1998f87d-25f7-4337-b794-c645289f9321 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.715892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:37.740746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.741596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.741881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.742156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.744005 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:37.744400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:37.745823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:37.760145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 195/399] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:37.761247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.768704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ced2b20-0871-4eda-b397-c9cf5e0826e5 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.811542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.813170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.813669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.814113 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.829068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:37.846255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:37.847476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:37.848598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:37.856354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 205/400] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:37.857455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:37.885923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30086519-a2d8-4034-ac23-4623e19ac309 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.922292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:37.923530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:37.923951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:37.924465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/groups/3085338af3414566ac41287ecb12462e/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:37.927134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] RBAC: Authorizing `identity:create_grant(project_id=696f929ba4b64cd88ceb4cdaa2f61d36, group_id=3085338af3414566ac41287ecb12462e, role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:37.928849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:37.968405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea10254-f537-4fde-ad25-dbaf83693383 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:37.983098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:37.998796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 196/401] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Jul 14 11:59:37 2022] PUT /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36/groups/3085338af3414566ac41287ecb12462e/roles/1ff35caa6d284df39cd287349f66c567 => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:37.998796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.014473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f37dd5-18d2-4e84-b163-51f737067bba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.015449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f37dd5-18d2-4e84-b163-51f737067bba None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.015782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f37dd5-18d2-4e84-b163-51f737067bba None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.016064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f37dd5-18d2-4e84-b163-51f737067bba None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.017554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f37dd5-18d2-4e84-b163-51f737067bba None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:38.019302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f37dd5-18d2-4e84-b163-51f737067bba None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:38.021473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 206/402] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:38.022799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f275f6-b205-45e1-99c1-f59f1d04f5d8 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.023109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f275f6-b205-45e1-99c1-f59f1d04f5d8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.023431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59f275f6-b205-45e1-99c1-f59f1d04f5d8 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.025401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 207/403] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:38.027337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f275f6-b205-45e1-99c1-f59f1d04f5d8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.047086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:38.140245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18af9fb9-3770-4267-b391-14c15bc8f041 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.164508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59f275f6-b205-45e1-99c1-f59f1d04f5d8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.187719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc12135-8887-4616-b53a-9a17838cded4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.188913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc12135-8887-4616-b53a-9a17838cded4 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.188913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc12135-8887-4616-b53a-9a17838cded4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.188913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc12135-8887-4616-b53a-9a17838cded4 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.189975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc12135-8887-4616-b53a-9a17838cded4 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:38.189975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc12135-8887-4616-b53a-9a17838cded4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:38.191606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc12135-8887-4616-b53a-9a17838cded4 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:38.200459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 197/404] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:38.201723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90085ad-9d9c-44a1-825e-d55e0c8011dc None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.202010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90085ad-9d9c-44a1-825e-d55e0c8011dc None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.202288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90085ad-9d9c-44a1-825e-d55e0c8011dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.207647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.208931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.209306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.209671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.210971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:38.212486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:38.214552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 208/405] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:38 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:38.215602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.221059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d90085ad-9d9c-44a1-825e-d55e0c8011dc None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:38.223429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d90085ad-9d9c-44a1-825e-d55e0c8011dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:38.270617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:38.295637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 198/406] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:38.304030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90085ad-9d9c-44a1-825e-d55e0c8011dc None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.330046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-741acc65-dd2c-4364-99c4-d9877011c00c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.350479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:38.380642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.382648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.382961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.383240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.384997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:38.387124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:38.389974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 209/407] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:38 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:38.391262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.443167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d90085ad-9d9c-44a1-825e-d55e0c8011dc None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.486075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.488809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.489323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.489799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e/groups/3085338af3414566ac41287ecb12462e/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.492153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] RBAC: Authorizing `identity:create_grant(project_id=5586dcebfe8c4e81bb952f00a518722e, group_id=3085338af3414566ac41287ecb12462e, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:38.494351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:38.521155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b95ac1-4442-46ce-94d9-965cdc8c18df None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.544159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:38.557021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.557824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.558109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.558389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.571111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:38.579438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:38.580340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:38.581150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 210/408] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:38.583610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.603368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 199/409] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Jul 14 11:59:38 2022] PUT /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e/groups/3085338af3414566ac41287ecb12462e/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:38.604371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.773412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8badb0ca-29ea-406b-868e-714a16e4b0ed None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.808676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:38.822925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3046a191-0597-476c-8475-6feb471ac71c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.824312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3046a191-0597-476c-8475-6feb471ac71c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.824598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3046a191-0597-476c-8475-6feb471ac71c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.824880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3046a191-0597-476c-8475-6feb471ac71c None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.826927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3046a191-0597-476c-8475-6feb471ac71c None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:38.828403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3046a191-0597-476c-8475-6feb471ac71c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:38.830354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 211/410] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:38 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:38.831720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b12363-947d-4d13-9313-98ebba0e64bf None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.831950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b12363-947d-4d13-9313-98ebba0e64bf None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.832224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b12363-947d-4d13-9313-98ebba0e64bf None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.833753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 212/411] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:38.837329 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b12363-947d-4d13-9313-98ebba0e64bf None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:38.844204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df31a435-7a68-426c-ade4-01a20e6ee68e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.892530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39cf30f-be95-4a43-bf9b-9750e891a96e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.893216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cf30f-be95-4a43-bf9b-9750e891a96e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.893601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cf30f-be95-4a43-bf9b-9750e891a96e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.893923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39cf30f-be95-4a43-bf9b-9750e891a96e None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.895388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39cf30f-be95-4a43-bf9b-9750e891a96e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:38.895777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39cf30f-be95-4a43-bf9b-9750e891a96e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:38.897285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39cf30f-be95-4a43-bf9b-9750e891a96e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:38.899829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:38.907565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 200/412] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 303 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:38.908869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c55acc-11bf-4ca6-90b8-0793eb98a9b6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.909167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c55acc-11bf-4ca6-90b8-0793eb98a9b6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.909450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c55acc-11bf-4ca6-90b8-0793eb98a9b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:38.927329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-14c55acc-11bf-4ca6-90b8-0793eb98a9b6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:38.928596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-14c55acc-11bf-4ca6-90b8-0793eb98a9b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:38.961518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b12363-947d-4d13-9313-98ebba0e64bf None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:38.997157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:38.997958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:38.998233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:38.998508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e/groups/3085338af3414566ac41287ecb12462e/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:39.000737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] RBAC: Authorizing `identity:create_grant(project_id=5586dcebfe8c4e81bb952f00a518722e, group_id=3085338af3414566ac41287ecb12462e, role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:39.002164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:39.002657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 201/413] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:39.007686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c55acc-11bf-4ca6-90b8-0793eb98a9b6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:39.074034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 213/414] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Jul 14 11:59:38 2022] PUT /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e/groups/3085338af3414566ac41287ecb12462e/roles/1ff35caa6d284df39cd287349f66c567 => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:39.080363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:39.118895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:39.165579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c55acc-11bf-4ca6-90b8-0793eb98a9b6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:39.227085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:39.228288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:39.228571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:39.228845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] PATH_INFO: `/v3/groups/13ab8d5d3af749f8b651e1af924066d9` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:39.231157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] RBAC: Authorizing `identity:get_group(group_id=13ab8d5d3af749f8b651e1af924066d9)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:39.232625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:39.234345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 202/415] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:39 2022] GET /identity/v3/groups/13ab8d5d3af749f8b651e1af924066d9 => generated 229 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:39.243868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:39.298396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22df936c-99f3-4b3d-a5dd-564de756438a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:39.308664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:39.340816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:39.341731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:39.342024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:39.342406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:39.354097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:39.362322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:39.363793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:39.364598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:39.369059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 214/416] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:39.370728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:39.392893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-296eb88a-8902-4424-9fd6-c494eeda683a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:39.444272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:39.445149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:39.445429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:39.445702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] PATH_INFO: `/v3/projects/1074c03ef1614e229d46654abff5220e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:39.447116 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] RBAC: Authorizing `identity:get_project(project_id=1074c03ef1614e229d46654abff5220e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:39.448553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:39.454988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 203/417] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:39 2022] GET /identity/v3/projects/1074c03ef1614e229d46654abff5220e => generated 344 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:39.461663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:39.488872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:39.534327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91dcc180-5308-4acd-ba1e-ddb3ca99173b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:39.599911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:39.600487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:39.600760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:39.601042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:39.601795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] RBAC: Adding query filter params (name=member) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:39.602106 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:39.603822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:39.630072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 215/418] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:39 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:39.633710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:39.647234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f4b1e4-e23f-4b37-aa1a-a4d33f16eedf None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:39.693706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:39.700402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:39.701581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:39.702026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:39.706939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] PATH_INFO: `/v3/projects/1074c03ef1614e229d46654abff5220e/groups/13ab8d5d3af749f8b651e1af924066d9/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:39.706939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] RBAC: Authorizing `identity:create_grant(project_id=1074c03ef1614e229d46654abff5220e, group_id=13ab8d5d3af749f8b651e1af924066d9, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:39.706939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:39.761158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 204/419] 10.222.0.27 () {58 vars in 1442 bytes} [Thu Jul 14 11:59:39 2022] PUT /identity/v3/projects/1074c03ef1614e229d46654abff5220e/groups/13ab8d5d3af749f8b651e1af924066d9/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:39.795526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96ca93b-4a23-4327-b8ad-2b80c274f21b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:39.836088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2cd581-abd4-4913-ae2c-8f4bbcbf89d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:39.837133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2cd581-abd4-4913-ae2c-8f4bbcbf89d9 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:39.837563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2cd581-abd4-4913-ae2c-8f4bbcbf89d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:39.837983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2cd581-abd4-4913-ae2c-8f4bbcbf89d9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:39.915595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2cd581-abd4-4913-ae2c-8f4bbcbf89d9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:39.917356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c2cd581-abd4-4913-ae2c-8f4bbcbf89d9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:39.967186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:40.030757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 216/420] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 14 11:59:40.039908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:40.127091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:40.154885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae56a972-57b8-4e3f-84d0-44c6c3db87b9 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:40.190657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:40.191474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fc3a67-a74e-4b4c-a649-2741fb6f2b92 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:40.191714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fc3a67-a74e-4b4c-a649-2741fb6f2b92 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:40.191906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42fc3a67-a74e-4b4c-a649-2741fb6f2b92 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:40.192016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:40.192290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:40.192559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:40.193411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 217/421] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:59:40.194526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:40.195579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:40.198983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8aa69-a0ec-4181-a80a-b4675ad3afd6 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:40.205141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8aa69-a0ec-4181-a80a-b4675ad3afd6 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:40.205141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b8aa69-a0ec-4181-a80a-b4675ad3afd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:40.205304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 205/422] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:40 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:40.205304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:40.217866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-84b8aa69-a0ec-4181-a80a-b4675ad3afd6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:40.219415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-84b8aa69-a0ec-4181-a80a-b4675ad3afd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:40.308283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c164cc6-9f11-43df-8422-a89a95cbbe7c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:40.345459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:40.346419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:40.346704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:40.347036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:40.348275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:40.350512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:40.356472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 206/423] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:40 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:40.357674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:40.361887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 218/424] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:40.371181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:40.399847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8aa69-a0ec-4181-a80a-b4675ad3afd6 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:40.464217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce28d230-c173-479b-94ed-238135a29eec None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:40.500303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:40.500963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:40.501245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:40.501520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:40.502470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:40.502876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:40.505317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:40.519092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 207/425] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:40 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 611 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:40.545773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:40.624202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b8aa69-a0ec-4181-a80a-b4675ad3afd6 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:40.677885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:40.679128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:40.679613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:40.680103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:40.696367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:40.705148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:40.706023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:40.708198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 219/426] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:40 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 308 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:40.717388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:40.809913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:40.837816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e3413c-c728-4602-a697-e886c33c90db None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:40.876169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7589b3-17d5-43a2-b08c-3b881798cf50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:40.876900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7589b3-17d5-43a2-b08c-3b881798cf50 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:40.877182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7589b3-17d5-43a2-b08c-3b881798cf50 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:40.877460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7589b3-17d5-43a2-b08c-3b881798cf50 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:40.878890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7589b3-17d5-43a2-b08c-3b881798cf50 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:40.879359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7589b3-17d5-43a2-b08c-3b881798cf50 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:40.880736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7589b3-17d5-43a2-b08c-3b881798cf50 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:40.891081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 208/427] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:40 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:40.897190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.016661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65ecc02-b02e-45ef-be8d-489ff6ae58a1 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.036126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:41.058377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.059056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.059342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.059620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.141214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:41.144846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:41.154136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ded22-0e2a-4ba7-a07c-fb89d976d89b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.154616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ded22-0e2a-4ba7-a07c-fb89d976d89b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.154731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-280ded22-0e2a-4ba7-a07c-fb89d976d89b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.156215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 209/428] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:41.162573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36ba247-5f80-4527-8551-b03e96e55df5 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.163233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36ba247-5f80-4527-8551-b03e96e55df5 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.163713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36ba247-5f80-4527-8551-b03e96e55df5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.189098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c36ba247-5f80-4527-8551-b03e96e55df5 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:41.190319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c36ba247-5f80-4527-8551-b03e96e55df5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:41.261321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 220/429] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:41.266605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.283106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 210/430] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:41.293265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:41.307244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36ba247-5f80-4527-8551-b03e96e55df5 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.394532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:41.401855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f3184-4cbe-48bb-b09b-42385f911b3b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.444250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36ba247-5f80-4527-8551-b03e96e55df5 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.446922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.448721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.449012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.449288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] PATH_INFO: `/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.450523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] RBAC: Authorizing `identity:get_project(project_id=696f929ba4b64cd88ceb4cdaa2f61d36)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:41.452034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:41.454058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 221/431] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:41 2022] GET /identity/v3/projects/696f929ba4b64cd88ceb4cdaa2f61d36 => generated 298 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:41.460163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.490340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.491607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.492110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.492592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] PATH_INFO: `/v3/roles/member` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.507287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:41.515640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:41.516536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 14 11:59:41.517355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:41.524946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 211/432] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:41 2022] GET /identity/v3/roles/member => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:41.526034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.594152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4f1c14-2a61-41da-a1f3-17c197702577 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.607185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:41.635404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e2326b-aa36-4e72-b290-ac83c963b9bf None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.638517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf87e72-bfe9-4282-826b-58efdb902634 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.638517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf87e72-bfe9-4282-826b-58efdb902634 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.638517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf87e72-bfe9-4282-826b-58efdb902634 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.638517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf87e72-bfe9-4282-826b-58efdb902634 None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.639899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf87e72-bfe9-4282-826b-58efdb902634 None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:41.641356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf87e72-bfe9-4282-826b-58efdb902634 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:41.647827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 222/433] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:41 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:41.647827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116f3566-df41-4f2b-a8fe-46d28f634636 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.647827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116f3566-df41-4f2b-a8fe-46d28f634636 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.647827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-116f3566-df41-4f2b-a8fe-46d28f634636 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.647827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 223/434] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:41.648762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116f3566-df41-4f2b-a8fe-46d28f634636 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.667828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:41.678786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fcaf37-9e4a-4c2b-a081-ae4f265a0067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.679589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fcaf37-9e4a-4c2b-a081-ae4f265a0067 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.679883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fcaf37-9e4a-4c2b-a081-ae4f265a0067 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.680398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fcaf37-9e4a-4c2b-a081-ae4f265a0067 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.681489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fcaf37-9e4a-4c2b-a081-ae4f265a0067 None admin] RBAC: Adding query filter params (name=member) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:41.681978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fcaf37-9e4a-4c2b-a081-ae4f265a0067 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:41.683393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0fcaf37-9e4a-4c2b-a081-ae4f265a0067 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:41.696402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 212/435] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 11:59:41 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:41.696402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52ae3a-efa1-4316-9360-52ecbab015b3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.696402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52ae3a-efa1-4316-9360-52ecbab015b3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.696402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f52ae3a-efa1-4316-9360-52ecbab015b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.712924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-8f52ae3a-efa1-4316-9360-52ecbab015b3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:41.714395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8f52ae3a-efa1-4316-9360-52ecbab015b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:41.770045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-116f3566-df41-4f2b-a8fe-46d28f634636 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.791231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 213/436] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:41.791231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f52ae3a-efa1-4316-9360-52ecbab015b3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.813339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.814110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.814391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.814803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.815754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:41.816602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:41.818171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:41.825025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 224/437] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:41 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=696f929ba4b64cd88ceb4cdaa2f61d36&role.id=1ff35caa6d284df39cd287349f66c567 => generated 611 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:41.825910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:41.853500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:41.928126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f52ae3a-efa1-4316-9360-52ecbab015b3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.948538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:41.955883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3d4b82-b9c9-4311-bfb7-9e22d0c12bdc None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:41.969685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.970720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.971246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.971974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:41.995279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:41.996095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:41.996380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:41.997541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.010235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:42.018562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:42.020453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:42.021235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:42.025920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 225/438] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:42.026583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.063643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:42.065131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:42.130522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.154344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d536bc60-bf69-44b4-a05b-576e77c5846a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.202916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:42.203973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.204490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.204611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 214/439] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:41 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:42.205416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.206514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:42.207104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:42.208733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:42.210524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.219655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 226/440] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:42.226402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.235347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.335510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bedfb26-dc68-4cef-a32a-663cfe6d2dab None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.345912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b21bfbbb-d6e7-4198-8045-b6fe5e035740 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.372910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:42.373776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.374058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.374355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.375523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:42.377028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:42.379433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 215/441] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:42.382259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8f4182-9ba0-4e59-92f3-61076544ba12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:42.382884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8f4182-9ba0-4e59-92f3-61076544ba12 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.383170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8f4182-9ba0-4e59-92f3-61076544ba12 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.383445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8f4182-9ba0-4e59-92f3-61076544ba12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.384660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.395730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.453846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8f4182-9ba0-4e59-92f3-61076544ba12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:42.455356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8f4182-9ba0-4e59-92f3-61076544ba12 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:42.469203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.483452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0629683d-bab3-403d-9913-18ad9da7f797 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.517344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de415c2d-c89c-4b0b-9f08-8266ace72ac5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:42.518195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de415c2d-c89c-4b0b-9f08-8266ace72ac5 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.518479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de415c2d-c89c-4b0b-9f08-8266ace72ac5 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.518836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de415c2d-c89c-4b0b-9f08-8266ace72ac5 None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.520188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de415c2d-c89c-4b0b-9f08-8266ace72ac5 None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:42.521641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de415c2d-c89c-4b0b-9f08-8266ace72ac5 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:42.523486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 216/442] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:42.524712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cb4441-bd24-4cc4-bfb2-d71dcc7606bb None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.524997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cb4441-bd24-4cc4-bfb2-d71dcc7606bb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.525271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cb4441-bd24-4cc4-bfb2-d71dcc7606bb None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.526798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 217/443] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:42.529083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cb4441-bd24-4cc4-bfb2-d71dcc7606bb None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.555408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 227/444] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 14 11:59:42.557789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b86be3-ed02-461d-87a4-3b19d81691d0 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.558150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b86be3-ed02-461d-87a4-3b19d81691d0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.558440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b86be3-ed02-461d-87a4-3b19d81691d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.574312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c9b86be3-ed02-461d-87a4-3b19d81691d0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:42.575526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c9b86be3-ed02-461d-87a4-3b19d81691d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:42.610231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.636378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cb4441-bd24-4cc4-bfb2-d71dcc7606bb None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.641402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 228/445] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:42.642221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b86be3-ed02-461d-87a4-3b19d81691d0 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.670220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:42.670913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.671201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.671481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.672316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=56f2092f7f5e42b09679dec166d174c5, scope.project.id=5586dcebfe8c4e81bb952f00a518722e) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:42.672629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:42.674075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:42.680162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 218/446] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=56f2092f7f5e42b09679dec166d174c5 => generated 611 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:42.680932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.701952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.764166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9b86be3-ed02-461d-87a4-3b19d81691d0 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.801807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ec90b4-037f-4bf8-ad20-c388204bf99a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.803784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:42.804683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.804949 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.805228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] PATH_INFO: `/v3/projects/5586dcebfe8c4e81bb952f00a518722e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.806460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] RBAC: Authorizing `identity:get_project(project_id=5586dcebfe8c4e81bb952f00a518722e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:42.808003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:42.809765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 229/447] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/projects/5586dcebfe8c4e81bb952f00a518722e => generated 302 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:42.815556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.844202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:42.845580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:42.845873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:42.846735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:42.859229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:42.861147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:42.868144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:42.869056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:42.869836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 219/448] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:42.875598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:42.940390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2db2a918-d47c-4cb1-8a47-99a022fd65ef None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:42.976161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:42.998839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:43.000191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:43.000191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:43.000377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] PATH_INFO: `/v3/groups/3085338af3414566ac41287ecb12462e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:43.002818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] RBAC: Authorizing `identity:get_group(group_id=3085338af3414566ac41287ecb12462e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:43.004426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:43.005845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 230/449] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/groups/3085338af3414566ac41287ecb12462e => generated 226 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:43.007436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dbe81f-dfe1-4ecc-a18b-1bf72961a90f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:43.012927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:43.066811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:43.067497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:43.067975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:43.068472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:43.069489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:43.070053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:43.071662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:43.084802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 220/450] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:43.090366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:43.134431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635f0953-931c-42fb-bcf9-c4399a92e222 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:43.141964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:43.169159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:43.169814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:43.170090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:43.170363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:43.171228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] RBAC: Adding query filter params (group.id=3085338af3414566ac41287ecb12462e, role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=5586dcebfe8c4e81bb952f00a518722e) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:43.171551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:43.173110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:43.181439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 231/451] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:43 2022] GET /identity/v3/role_assignments?group.id=3085338af3414566ac41287ecb12462e&scope.project.id=5586dcebfe8c4e81bb952f00a518722e&role.id=1ff35caa6d284df39cd287349f66c567 => generated 611 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:43.216228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69c03f0c-c22f-431e-97b8-6668ccbb8075 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:43.253774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:43.254448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:43.254732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:43.255036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:43.268125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:43.330069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:43.331747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:43.431913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 221/452] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:43 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:43.437507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:43.542882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c78af424-cfc7-4455-b44e-fa75aa6b322d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:43.575368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:43.576213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:43.576486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:43.576753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] PATH_INFO: `/v3/projects/1074c03ef1614e229d46654abff5220e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:43.577972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] RBAC: Authorizing `identity:get_project(project_id=1074c03ef1614e229d46654abff5220e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:43.579461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:43.581121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 232/453] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 11:59:43 2022] GET /identity/v3/projects/1074c03ef1614e229d46654abff5220e => generated 344 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:43.586679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:43.626760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:43.682195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221d21f2-2242-4dd7-9575-2621262efda0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:43.714141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92aecd81-2219-4634-9e35-e448ca941a30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:43.715016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aecd81-2219-4634-9e35-e448ca941a30 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:43.715293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aecd81-2219-4634-9e35-e448ca941a30 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:43.715563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92aecd81-2219-4634-9e35-e448ca941a30 None admin] PATH_INFO: `/v3/groups/13ab8d5d3af749f8b651e1af924066d9` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:43.716711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92aecd81-2219-4634-9e35-e448ca941a30 None admin] RBAC: Authorizing `identity:get_group(group_id=13ab8d5d3af749f8b651e1af924066d9)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:43.718167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92aecd81-2219-4634-9e35-e448ca941a30 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:43.719776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 222/454] 10.222.0.27 () {56 vars in 1180 bytes} [Thu Jul 14 11:59:43 2022] GET /identity/v3/groups/13ab8d5d3af749f8b651e1af924066d9 => generated 229 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:43.726199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:43.765703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:43.823386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24641bfd-f0bd-4bc7-8636-6bae94ba0b4b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:43.855829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9abf7e40-f0fd-4cd9-bd7e-57e8ebd852cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:43.856469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf7e40-f0fd-4cd9-bd7e-57e8ebd852cb None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:43.856753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf7e40-f0fd-4cd9-bd7e-57e8ebd852cb None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:43.857031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9abf7e40-f0fd-4cd9-bd7e-57e8ebd852cb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:43.857859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf7e40-f0fd-4cd9-bd7e-57e8ebd852cb None admin] RBAC: Adding query filter params (group.id=13ab8d5d3af749f8b651e1af924066d9, role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=1074c03ef1614e229d46654abff5220e) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:43.858171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf7e40-f0fd-4cd9-bd7e-57e8ebd852cb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:43.859633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9abf7e40-f0fd-4cd9-bd7e-57e8ebd852cb None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:43.865831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 233/455] 10.222.0.27 () {56 vars in 1376 bytes} [Thu Jul 14 11:59:43 2022] GET /identity/v3/role_assignments?group.id=13ab8d5d3af749f8b651e1af924066d9&scope.project.id=1074c03ef1614e229d46654abff5220e&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 611 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:46.227704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0bdef3-0361-4f2f-9e45-bc2ba42404c1 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.228561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0bdef3-0361-4f2f-9e45-bc2ba42404c1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.229033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f0bdef3-0361-4f2f-9e45-bc2ba42404c1 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.231110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 223/456] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:46.235986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d640e4-0851-4822-9cd9-8c710421e624 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.236411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d640e4-0851-4822-9cd9-8c710421e624 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.236540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d640e4-0851-4822-9cd9-8c710421e624 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.263952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-11d640e4-0851-4822-9cd9-8c710421e624 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:46.265414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-11d640e4-0851-4822-9cd9-8c710421e624 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:46.361050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 234/457] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:46.398308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0bdef3-0361-4f2f-9e45-bc2ba42404c1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:46.408549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b5f9f-35dd-4500-87dc-9fc25ed7051b None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.410752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b5f9f-35dd-4500-87dc-9fc25ed7051b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.411442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83b5f9f-35dd-4500-87dc-9fc25ed7051b None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.414570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 235/458] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:46 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 11:59:46.420070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c982b9a1-2535-410b-812c-b7abcd1c9fa7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.420602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c982b9a1-2535-410b-812c-b7abcd1c9fa7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.421023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c982b9a1-2535-410b-812c-b7abcd1c9fa7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.446653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c982b9a1-2535-410b-812c-b7abcd1c9fa7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:46.447881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c982b9a1-2535-410b-812c-b7abcd1c9fa7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:46.503827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:46.571352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f0bdef3-0361-4f2f-9e45-bc2ba42404c1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:46.584954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:46.593003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 236/459] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:46.621401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:46.622483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.624828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.624828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.634111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:46.636081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:46.663961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c982b9a1-2535-410b-812c-b7abcd1c9fa7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:46.665112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:46.665947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:46.666829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:46.666829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:46.666829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:46.666829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:46.668831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 224/460] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:46.674115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:46.855317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450320fc-766e-4a6b-a0cb-7680488e1434 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:46.856937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c982b9a1-2535-410b-812c-b7abcd1c9fa7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:46.899334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:46.900161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.900425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.900703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.902452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330ec82-6360-4dfe-9561-ef8d33907779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:46.903940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ec82-6360-4dfe-9561-ef8d33907779 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.904425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ec82-6360-4dfe-9561-ef8d33907779 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.904913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ec82-6360-4dfe-9561-ef8d33907779 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.905843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330ec82-6360-4dfe-9561-ef8d33907779 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:46.906348 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330ec82-6360-4dfe-9561-ef8d33907779 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:46.908433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330ec82-6360-4dfe-9561-ef8d33907779 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:46.911879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:46.913333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:46.931174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 225/461] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:46.932802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e710413-e675-4371-b040-d8ff3a3556fd None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:46.933141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:46.933924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:46.934750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:46.934750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:46.934750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:46.934750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:46.934750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 237/462] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:46.935920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e710413-e675-4371-b040-d8ff3a3556fd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:46.936429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e710413-e675-4371-b040-d8ff3a3556fd None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:46.938268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 226/463] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:46 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:46.940988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:46.942242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e710413-e675-4371-b040-d8ff3a3556fd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:46.960532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:46.967215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:47.051628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e710413-e675-4371-b040-d8ff3a3556fd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.059938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab31c97-afa7-4248-ac5d-a604144d8fee None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.085572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f7b7f4-8cf6-4261-8a72-7173477a2043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.086247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f7b7f4-8cf6-4261-8a72-7173477a2043 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.086523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f7b7f4-8cf6-4261-8a72-7173477a2043 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.086821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f7b7f4-8cf6-4261-8a72-7173477a2043 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.087536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f7b7f4-8cf6-4261-8a72-7173477a2043 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:47.087839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f7b7f4-8cf6-4261-8a72-7173477a2043 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.089185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f7b7f4-8cf6-4261-8a72-7173477a2043 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:47.099314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73626814-3f7e-49f7-9cec-18ba34fa2be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.100218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73626814-3f7e-49f7-9cec-18ba34fa2be2 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.100680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73626814-3f7e-49f7-9cec-18ba34fa2be2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.101135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73626814-3f7e-49f7-9cec-18ba34fa2be2 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.101536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 227/464] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:47.102427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73626814-3f7e-49f7-9cec-18ba34fa2be2 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.102544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb30b07a-957e-47b3-b134-53f376d6f5f9 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.102792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb30b07a-957e-47b3-b134-53f376d6f5f9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.103078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb30b07a-957e-47b3-b134-53f376d6f5f9 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.104513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73626814-3f7e-49f7-9cec-18ba34fa2be2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:47.104622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 228/465] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:47.105624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f61d6-72ae-4f1a-a504-07bbc2d5253a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.105846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f61d6-72ae-4f1a-a504-07bbc2d5253a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.106119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3f61d6-72ae-4f1a-a504-07bbc2d5253a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.126367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1e3f61d6-72ae-4f1a-a504-07bbc2d5253a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:47.127686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1e3f61d6-72ae-4f1a-a504-07bbc2d5253a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:47.200830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 229/466] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:47.200830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f61d6-72ae-4f1a-a504-07bbc2d5253a None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:47.201097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 238/467] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:46 2022] POST /identity/v3/users => generated 246 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:47.201097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0705bf20-30a1-4928-b096-6de5159a43cd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.201097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0705bf20-30a1-4928-b096-6de5159a43cd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.201097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0705bf20-30a1-4928-b096-6de5159a43cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.217655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0705bf20-30a1-4928-b096-6de5159a43cd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:47.218999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0705bf20-30a1-4928-b096-6de5159a43cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:47.226914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:47.321159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:47.332517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 239/468] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:47.332517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0705bf20-30a1-4928-b096-6de5159a43cd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:47.377379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3f61d6-72ae-4f1a-a504-07bbc2d5253a None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.420823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.421813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.421813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.422056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.424114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.424233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:47.468909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0705bf20-30a1-4928-b096-6de5159a43cd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.506062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.506883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.507170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.507445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.518147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.519816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 230/469] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:47 2022] POST /identity/v3/users => generated 248 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:47.520040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:47.521434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:47.540836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:47.541773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:47.547696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:47.547696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:47.547696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:47.547696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:47.547696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 240/470] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:47.551753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:47.610005 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:47.635347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:47.653660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24114ff9-8d92-479a-b5b5-b0a522c270d7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.695823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338d5ecd-d41f-4f10-8702-50c9f414844f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.704904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.705829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.707195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.707195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.720287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.722595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:47.744427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:47.745343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:47.746541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:47.746541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:47.746541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:47.746541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:47.746541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 231/471] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:47.751932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:47.759785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.760733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.761019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.761312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.762566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:47.762908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.764753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:47.782859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 241/472] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:47.790803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:47.899468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee64cc2-037a-46f7-b7dd-739e5d9e02ab None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.918016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:47.924280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014cff53-c663-421e-bc46-727a5864daf8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:47.939053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.940132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.940658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.941138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.941539 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:47.942462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:47.943000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.944600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:47.959965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 232/473] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:47.966626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:47.967966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5dd0b2-153a-49af-bf62-f8c86179a4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:47.968647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5dd0b2-153a-49af-bf62-f8c86179a4f8 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:47.968932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5dd0b2-153a-49af-bf62-f8c86179a4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:47.969212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5dd0b2-153a-49af-bf62-f8c86179a4f8 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:47.969938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5dd0b2-153a-49af-bf62-f8c86179a4f8 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:47.971675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5dd0b2-153a-49af-bf62-f8c86179a4f8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:48.072641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 242/474] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:47 2022] POST /identity/v3/users => generated 248 bytes in 285 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:48.080338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb0e73-56e6-4bcb-81f1-fac8269f2dd6 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:48.080644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb0e73-56e6-4bcb-81f1-fac8269f2dd6 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:48.080909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92bb0e73-56e6-4bcb-81f1-fac8269f2dd6 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:48.082522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 243/475] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:48 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:48.089711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf6e37-894b-4c60-88b3-84e168a298a2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:48.090047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf6e37-894b-4c60-88b3-84e168a298a2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:48.090278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddf6e37-894b-4c60-88b3-84e168a298a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:48.095284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4d00669-589d-41dd-9ea2-974ba413e963 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:48.118306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3ddf6e37-894b-4c60-88b3-84e168a298a2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:48.120213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3ddf6e37-894b-4c60-88b3-84e168a298a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:48.137994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:48.138852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:48.139204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:48.139549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:48.140435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:48.141992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:48.204262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 244/476] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:48.227603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 233/477] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:47 2022] POST /identity/v3/users => generated 249 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:48.242324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf6e37-894b-4c60-88b3-84e168a298a2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:48.377273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:48.382657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddf6e37-894b-4c60-88b3-84e168a298a2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:48.426053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:48.427137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:48.427613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:48.428087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:48.429099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:48.430806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:48.464400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.sql.core [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 14 11:59:48.464400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 14 11:59:48.464400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [parameters: {'id': '599a15d3f368425a9f04ea4a3f570e33', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 14 11:59:48.464400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80487) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result = self._query(query) Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application conn.query(q) Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result.read() Jul 14 11:59:48.481474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return ref.to_dict() Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 14 11:59:48.482525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application next(self.gen) Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application yield resource Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application next(self.gen) Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.session.rollback() Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.force_reraise() Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise self.value Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application session.commit() Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 14 11:59:48.483655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._prepare_impl() Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.session.flush() Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._flush(objects) Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application compat.raise_( Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application flush_context.execute() Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rec.execute(self) Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 14 11:59:48.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 14 11:59:48.485904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result = self._query(query) Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application conn.query(q) Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result.read() Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 14 11:59:48.488413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [parameters: {'id': '599a15d3f368425a9f04ea4a3f570e33', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:48.490560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 14 11:59:48.492525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:48.494425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 245/478] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Jul 14 11:59:48 2022] POST /identity/v3/roles => generated 144 bytes in 245 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:48.494559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:48.540478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:48.704029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c34ef6-ea38-4f03-a52b-77a717d96051 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:48.745748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49630bfa-61c7-4a00-a3a4-8e743af22a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:48.746572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49630bfa-61c7-4a00-a3a4-8e743af22a28 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:48.746880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49630bfa-61c7-4a00-a3a4-8e743af22a28 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:48.747158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49630bfa-61c7-4a00-a3a4-8e743af22a28 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:48.758010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49630bfa-61c7-4a00-a3a4-8e743af22a28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-49630bfa-61c7-4a00-a3a4-8e743af22a28 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:48.770952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:48.771877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:48.773086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 234/479] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 11:59:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:48.778280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:48.902194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6a2ecc-d60b-4553-88a7-a74481cecc85 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:48.942374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c83a0ddf-cf47-4ea9-8545-a921c75a93ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:48.943424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83a0ddf-cf47-4ea9-8545-a921c75a93ec None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:48.943920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83a0ddf-cf47-4ea9-8545-a921c75a93ec None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:48.944400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c83a0ddf-cf47-4ea9-8545-a921c75a93ec None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:48.945381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83a0ddf-cf47-4ea9-8545-a921c75a93ec None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:48.945890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83a0ddf-cf47-4ea9-8545-a921c75a93ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:48.947490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c83a0ddf-cf47-4ea9-8545-a921c75a93ec None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:48.960474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 246/480] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:49.495494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00155580-749e-4171-b43b-a1a2b16fec97 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:49.496579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00155580-749e-4171-b43b-a1a2b16fec97 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:49.497088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00155580-749e-4171-b43b-a1a2b16fec97 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:49.498976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 235/481] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:49.504817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7bc4c-6244-4dec-822a-9316238b219f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:49.505243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7bc4c-6244-4dec-822a-9316238b219f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:49.505437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20b7bc4c-6244-4dec-822a-9316238b219f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:49.526420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-20b7bc4c-6244-4dec-822a-9316238b219f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:49.527950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-20b7bc4c-6244-4dec-822a-9316238b219f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:49.547515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:49.615330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 247/482] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:49.638642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00155580-749e-4171-b43b-a1a2b16fec97 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:49.771339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:49.783484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00155580-749e-4171-b43b-a1a2b16fec97 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:49.835447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:49.836342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:49.836626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:49.836903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:49.852915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:49.860685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:49.861564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:49.862377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 236/483] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:49.868297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b7bc4c-6244-4dec-822a-9316238b219f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:50.015584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20b7bc4c-6244-4dec-822a-9316238b219f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:50.063286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:50.063868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.064145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.064413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.065156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:50.065485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:50.067291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:50.077958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 248/484] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:50.083694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:50.236308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b8e877e-d983-40df-bf94-594f79947437 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:50.280975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720d49e5-935c-4574-96fa-bb64922f62ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:50.281774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d49e5-935c-4574-96fa-bb64922f62ea None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.282054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d49e5-935c-4574-96fa-bb64922f62ea None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.282331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d49e5-935c-4574-96fa-bb64922f62ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.300858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720d49e5-935c-4574-96fa-bb64922f62ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:50.302326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720d49e5-935c-4574-96fa-bb64922f62ea None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-720d49e5-935c-4574-96fa-bb64922f62ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:50.323254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:50.324111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:50.324947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:50.324947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:50.324947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:50.324947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:50.324947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 237/485] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:50.331505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:50.351333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:50.377655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a90b81-7330-42ab-88d0-3b3a9acb3aa4 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.378295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a90b81-7330-42ab-88d0-3b3a9acb3aa4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.378812 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a90b81-7330-42ab-88d0-3b3a9acb3aa4 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.380389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 238/486] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:50.385610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d8430-21cf-4d90-9f71-56a1515e7a9f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.386156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d8430-21cf-4d90-9f71-56a1515e7a9f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.386664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39d8430-21cf-4d90-9f71-56a1515e7a9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.399374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:50.405254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d39d8430-21cf-4d90-9f71-56a1515e7a9f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:50.406796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d39d8430-21cf-4d90-9f71-56a1515e7a9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:50.478375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 239/487] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:50.492187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aa169d-7dd7-4517-9a82-ba34662a0609 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:50.514043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d8430-21cf-4d90-9f71-56a1515e7a9f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:50.529071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:50.530317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.530723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.531240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.532083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:50.532562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:50.534160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:50.546693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 249/488] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:50.552397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:50.665877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449436ad-589c-446c-9d30-83013b792fa0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:50.668628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:50.702961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39d8430-21cf-4d90-9f71-56a1515e7a9f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:50.711021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:50.712127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.712536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.712940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] PATH_INFO: `/v3/users/nova` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.721394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:50.724255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 11:59:50.744818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:50.745743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:50.746666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 250/489] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 11:59:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:50.747824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:50.748869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.749360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.749846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.751716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:50.762011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:50.768168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:50.771768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:50.772677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:50.773460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 240/490] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:50.781024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:50.936259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53713b8-dddc-4bcb-81fc-61f760a983cd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:50.941088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064f45a4-9e09-40b1-b898-f7c9b208d5ec None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:50.981636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:50.982144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:50.982425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:50.982700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:50.983579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:50.983885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:50.985315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:50.997309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.001460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8af835-2573-49f8-9684-25e12cea2b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.002142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8af835-2573-49f8-9684-25e12cea2b49 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.002509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8af835-2573-49f8-9684-25e12cea2b49 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.007874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8af835-2573-49f8-9684-25e12cea2b49 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.007874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8af835-2573-49f8-9684-25e12cea2b49 None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:51.007874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8af835-2573-49f8-9684-25e12cea2b49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:51.007874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8af835-2573-49f8-9684-25e12cea2b49 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:51.017216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 241/491] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:51.017216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5efb8-a07b-4f99-aa24-1ef0122b63c8 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.017216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5efb8-a07b-4f99-aa24-1ef0122b63c8 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.017216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5efb8-a07b-4f99-aa24-1ef0122b63c8 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.017216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 242/492] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:51.018944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a4688-8437-44ab-bbb3-beacb7e5519f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.019246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a4688-8437-44ab-bbb3-beacb7e5519f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.019704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a4688-8437-44ab-bbb3-beacb7e5519f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.021063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 243/493] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:51.023219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abb25ff-25e8-49c7-ae5e-151fcb65e934 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.023509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abb25ff-25e8-49c7-ae5e-151fcb65e934 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.023792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abb25ff-25e8-49c7-ae5e-151fcb65e934 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.032142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 251/494] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 11:59:50 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:51.035893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.042607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3abb25ff-25e8-49c7-ae5e-151fcb65e934 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:51.043833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3abb25ff-25e8-49c7-ae5e-151fcb65e934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:51.101881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.122792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 244/495] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:51.122792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f57e6-ee9f-44d7-8a1e-7a0f9e076282 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.122792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f57e6-ee9f-44d7-8a1e-7a0f9e076282 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.122792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641f57e6-ee9f-44d7-8a1e-7a0f9e076282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.144240 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-641f57e6-ee9f-44d7-8a1e-7a0f9e076282 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:51.145449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-641f57e6-ee9f-44d7-8a1e-7a0f9e076282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:51.155844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6a5ae84-aeeb-4f25-8458-82dc5a722574 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.191747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.192456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.192746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.193022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.204107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:51.205568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:51.216179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 245/496] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:51.216856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f57e6-ee9f-44d7-8a1e-7a0f9e076282 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:51.233529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:51.234406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:51.235288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:51.235288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:51.235288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:51.235288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:51.236474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 252/497] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:51.237509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.288064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.326333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641f57e6-ee9f-44d7-8a1e-7a0f9e076282 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.343936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d75643b-5a1f-46cb-ab85-ffc60d61675a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.349019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.368108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.369034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.369319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.369756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.378586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.379844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.380131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.380409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.392595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:51.401565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:51.402718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:51.403633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 253/498] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:51.404352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.449885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:51.451748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:51.513536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30e3045-bd95-42bc-97bb-f7c910451edd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.521190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.551405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.552070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.552338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.552790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.553673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:51.553984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:51.555408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:51.562663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 246/499] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:51.562663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.570308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 254/500] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:51.571177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.586141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.671873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bf25a5-0fa2-4c1a-8854-f574dcd9ce90 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.678163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309b4554-4286-4cce-af2b-a962574d405f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.708851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.709564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.709845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.710114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.716888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.717552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.717833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.718107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.718885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:51.719192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:51.720561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:51.725313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:51.729608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 255/501] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:51.730403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:51.734299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:51.735388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 247/502] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:51.736187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.760645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.813414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:51.832563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5963e4f8-ab72-481c-a385-ed6625c529cc None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.835055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0d4aec-4899-4f5d-b9d1-3331df25733e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:51.867634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.868411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.868691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.868966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.869187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc9d63-f24c-413a-b346-0eb0dc4dff1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:51.869969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc9d63-f24c-413a-b346-0eb0dc4dff1a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.870250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc9d63-f24c-413a-b346-0eb0dc4dff1a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.870524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc9d63-f24c-413a-b346-0eb0dc4dff1a None admin] PATH_INFO: `/v3/users/glance` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.878219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc9d63-f24c-413a-b346-0eb0dc4dff1a None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:51.879135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:51.879701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc9d63-f24c-413a-b346-0eb0dc4dff1a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:51.880573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-f0fc9d63-f24c-413a-b346-0eb0dc4dff1a None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 11:59:51.894726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:51.895841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:51.896777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 248/503] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/users/glance => generated 84 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:51.897654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb40e8d-64b5-4235-9d39-dc03c2931733 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:51.897748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb40e8d-64b5-4235-9d39-dc03c2931733 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:51.897903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:51.898660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:51.899795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:51.899795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:51.899795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:51.899795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:51.900002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb40e8d-64b5-4235-9d39-dc03c2931733 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:51.903032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 249/505] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:51.903216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 256/505] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:51.903890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.904301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb40e8d-64b5-4235-9d39-dc03c2931733 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:51.984120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.003175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcab4f49-c6c2-4e8c-a912-19ea4abadb81 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.007960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb40e8d-64b5-4235-9d39-dc03c2931733 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.037563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4de421-a9e2-4ef3-906a-4dc270cda203 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.038380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4de421-a9e2-4ef3-906a-4dc270cda203 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.038662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4de421-a9e2-4ef3-906a-4dc270cda203 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.038970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4de421-a9e2-4ef3-906a-4dc270cda203 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.041592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.042246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.042536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.042847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.043596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:52.043908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.045296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.049240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4de421-a9e2-4ef3-906a-4dc270cda203 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.050834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.051863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd4de421-a9e2-4ef3-906a-4dc270cda203 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.054395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 250/506] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:52.055225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-fd4de421-a9e2-4ef3-906a-4dc270cda203 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:52.070565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:52.071509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:52.072339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:52.072339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:52.072339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:52.072339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:52.072339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 257/507] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:52.073372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8eb567-a9cc-46c6-9a54-3a611398d5d6 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.073659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8eb567-a9cc-46c6-9a54-3a611398d5d6 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.073934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b8eb567-a9cc-46c6-9a54-3a611398d5d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.090544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1b8eb567-a9cc-46c6-9a54-3a611398d5d6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:52.091783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1b8eb567-a9cc-46c6-9a54-3a611398d5d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:52.156405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a02535-ebe8-4b78-8a81-d426ef7cf622 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.159481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 258/508] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:52.160567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8eb567-a9cc-46c6-9a54-3a611398d5d6 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.191575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.192250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.192536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.192816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.193657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:52.193983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.195441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.224739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.231107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 251/509] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:52.231932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.266596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b8eb567-a9cc-46c6-9a54-3a611398d5d6 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.290839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.300809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.301467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.301749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.302018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.302728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:52.303101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.304495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.320389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 259/510] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:52.320389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.337644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca30840a-4a39-4d59-b662-d2fc12e05c5b None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.374934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.375717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.375999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.376281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.386921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.388386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:52.406398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:52.407292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:52.408144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:52.408144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:52.408144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:52.408144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:52.408144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 252/511] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:52.408400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.419552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737065fe-1e92-4bee-8cc1-4656a735e003 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.441264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.452612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.453266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.453546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.453823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.454538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:52.454877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.456958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.468515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 260/512] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:52.469302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.508181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f541b4-d139-4a92-85f4-68cf6717eaec None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.518172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.542555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.543347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.543631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.543909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.554814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.556282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:52.573727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:52.575138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:52.575967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:52.575967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:52.575967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:52.575967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:52.575967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 253/513] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:52.575967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.613695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb9fcfd-110c-44e1-9f74-0e3218038c90 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.647554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.648198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.648479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.648756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.679735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf15d972-7c14-40d0-85f2-42ed9f8d93cf None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.696953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.712469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.713330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.713613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.713885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.715099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.717033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.720841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.722876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 11:59:52.739755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:52.740753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 254/514] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:52.741786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.764900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.821442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 261/515] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:52.822205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.845082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdd64a7-a1cb-4bf1-949e-ff825182770e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.879600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.880257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.880541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.880821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.881541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:52.881851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.883248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.895674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 255/516] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:52.896463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.904606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.922116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6343bfd-e635-4780-8df2-e85341c2b413 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:52.957539 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:52.958341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:52.958625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:52.958943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:52.965819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:52.967327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:52.978351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 11:59:52.981195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:52.982128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 11:59:52.983085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:52.987594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 262/517] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/users/cinder => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:52.988347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:52.996674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f3d0e9-b317-4d0c-89b7-f33a05bf70e1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.031668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.032333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.032617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.032898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.033620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:53.033935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.036211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.048005 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 256/518] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:53.048802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.093483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc08361-3bfc-4d95-8eb4-1540c12e6aef None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.126378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.127057 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.127346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.127624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.129226 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:53.129540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.131010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.146265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7095ad4-6ec3-4fa5-86c9-afb0632ff359 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.148133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 263/519] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 11:59:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:53.148936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.151715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:53.185427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.186218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.186502 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.186800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.197422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.198939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.200747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:53.216928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:53.217770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:53.218606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:53.218606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:53.218606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:53.218606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:53.218606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 257/520] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:53.219108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.249165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683d4815-b65e-448b-8a08-f999a5f9ce99 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.282157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.282973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.283254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.283525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.291518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.292959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 11:59:53.306941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:53.307969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 264/521] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/users/neutron => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:53.308897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.318801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a304aac-003c-48de-a203-2c5fd5c18b57 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.354179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.354828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.355115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.355391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.356223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:53.356534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.357980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.369094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:53.380328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 258/522] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:53.380328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.413934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b3b865-34ce-4221-9480-2e8cdeec1c5f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.429536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:53.447869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.448533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.448815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.449089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.449789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.451329 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.476056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61476b59-a73e-4ba1-9937-6f419d9264ee None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.509562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.510194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.510469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.510743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.511588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=da24a7ecdcc249f0820add0404a7856c) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:53.511900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.513314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.518825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 259/523] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/role_assignments?user.id=da24a7ecdcc249f0820add0404a7856c&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:53.519585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.541242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 265/524] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 11:59:53 2022] POST /identity/v3/users => generated 247 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 11:59:53.542147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.672132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b623dd0-eb2b-4ea0-81dd-fa319ac23824 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.681173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc8454-8262-48a1-80fd-6d760a9b3b21 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.695750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:53.704879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:53.721953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.722731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.723074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.726499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.726499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:53.726499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.726499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.739628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.743160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.743160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.743160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.743160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:53.743160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:53.743550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:53.748848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 266/525] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:53.749820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.757064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 260/526] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:53.757932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:53.871656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dca8707-604d-42df-b188-c54e25cceefc None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.890365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134ef14e-42cf-4440-a920-b082b0895046 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:53.924391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.925059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.925353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.925657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.929126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:53.930004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:53.930434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:53.930986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:53.980773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:53.985999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:54.009391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.011255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.019466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.020969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.134585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 261/527] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:54.140356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.140572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 267/528] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:54.141600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.224892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:54.250343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36abe0e5-3030-45a1-89e5-98ea83424fe2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.259152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:54.269541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e45b889-86cd-44ab-a75b-85ff318e0c8d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.289879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.291038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.291545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.292038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.303575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.305333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.312320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.313115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.313399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.313676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.322537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.324037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:54.326952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:54.327782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:54.328615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:54.328615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:54.328615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:54.328615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:54.331018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 268/529] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:54.332107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 11:59:54.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:54.345651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 262/530] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:54.347704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.469269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9304b1-fc2e-46d7-8dba-c1db8446b883 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.474367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc8510c6-972c-426a-98a7-8f5e9e56bc2b None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.497180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:54.506001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.507135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.507420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.507698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.512408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.517548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.517548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.517548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.517548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:54.517548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.517548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.518202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.520532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:54.520641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.535877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 263/531] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:54.536708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:54.543337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:54.544178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:54.545004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:54.545004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:54.545004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:54.545004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:54.552278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 269/532] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:54.553476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.666193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d3ee38-27b9-470d-89fa-295a5984e833 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.667139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d17212-6020-4dca-8f4b-290a929ef8fd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.710278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.711898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.712550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.712665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.712828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.713106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.713529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.713899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:54.714209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.714403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.717955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.728725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.730410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.741047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 264/533] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:54.742009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 11:59:54.748957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:54.749777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:54.750609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:54.753854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 270/534] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:54.754994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.805379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:54.828238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:54.873151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6117efd6-8174-4a1f-bc59-e5e1c7f62bb4 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.886604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad01ce1-f485-4625-9c73-d6a99e559856 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:54.910940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.911992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.912481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.912967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.914095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=86ace7e877e3484b8853536b77c76885) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:54.914635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.916451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.923116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 271/535] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/role_assignments?user.id=86ace7e877e3484b8853536b77c76885&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 259 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:54.924215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:54.940567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:54.941742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:54.942235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:54.942715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:54.943708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:54.944233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:54.946226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:54.973481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 265/536] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:54.979147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:55.123630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e01fcc-63ed-4198-854c-e9277789d7af None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:55.125955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:55.156735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf14e40-a2b9-422f-98be-e4bfff7fc40c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:55.173889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:55.174696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.175188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.175503 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.176259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:55.176568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:55.178005 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:55.191408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:55.202699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 272/537] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:55.209008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:55.226410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:55.227256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.227538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.227809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.239608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:55.241074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 11:59:55.267206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:55.268101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:55.269480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:55.270564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 266/538] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 291 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:55.279715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:55.386597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e44f52-6ecf-4351-bfd4-3a799dd99581 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:55.422730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:55.423427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.423713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.423984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.425528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:55.425806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:55.427301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:55.439933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 267/539] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 11:59:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:55.446024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:55.446887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad1ab0f-7d81-40f4-96be-27dda144ea94 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:55.495185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:55.535576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd516f1-57a2-40f6-8e44-56cd9e62e734 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:55.536246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd516f1-57a2-40f6-8e44-56cd9e62e734 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.536531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd516f1-57a2-40f6-8e44-56cd9e62e734 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.536811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd516f1-57a2-40f6-8e44-56cd9e62e734 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.537725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd516f1-57a2-40f6-8e44-56cd9e62e734 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=a4fff90ae5434b58932f468ca9c827f3) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:55.538036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd516f1-57a2-40f6-8e44-56cd9e62e734 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:55.539560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd516f1-57a2-40f6-8e44-56cd9e62e734 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:55.554374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e415a653-0f09-45c9-99db-1269fa788cc7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:55.564879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 273/540] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:55 2022] GET /identity/v3/role_assignments?user.id=a4fff90ae5434b58932f468ca9c827f3&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 259 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:55.593770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e6ff696-3eaa-4fb2-9ff7-66bc2c077d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:55.594456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ff696-3eaa-4fb2-9ff7-66bc2c077d54 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.594743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ff696-3eaa-4fb2-9ff7-66bc2c077d54 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.595062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e6ff696-3eaa-4fb2-9ff7-66bc2c077d54 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.596014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6ff696-3eaa-4fb2-9ff7-66bc2c077d54 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=e114b292a7f544b0b24016199edcef25) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:55.596349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6ff696-3eaa-4fb2-9ff7-66bc2c077d54 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:55.597881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e6ff696-3eaa-4fb2-9ff7-66bc2c077d54 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:55.604795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 268/541] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:55 2022] GET /identity/v3/role_assignments?user.id=e114b292a7f544b0b24016199edcef25&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:55.702843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6bbd9-6bd6-4ed9-9c33-e12cdd4a9cf8 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.703186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6bbd9-6bd6-4ed9-9c33-e12cdd4a9cf8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.703420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a6bbd9-6bd6-4ed9-9c33-e12cdd4a9cf8 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.704881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 274/542] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:55.724988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b42bb3d-4b9f-4bda-b827-1becccff2b95 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.725508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b42bb3d-4b9f-4bda-b827-1becccff2b95 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.725582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b42bb3d-4b9f-4bda-b827-1becccff2b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.752900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7b42bb3d-4b9f-4bda-b827-1becccff2b95 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:55.754176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7b42bb3d-4b9f-4bda-b827-1becccff2b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:55.837692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 269/543] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:55.860336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6bbd9-6bd6-4ed9-9c33-e12cdd4a9cf8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:55.890701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:55.922479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92529069-0ad9-449b-92f4-16d3b6033a74 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.923119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92529069-0ad9-449b-92f4-16d3b6033a74 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.923606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92529069-0ad9-449b-92f4-16d3b6033a74 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.927355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 270/544] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:55 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:55.931942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3486e3c6-4fd1-4612-b57a-d70f0d9ebeb0 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:55.933368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3486e3c6-4fd1-4612-b57a-d70f0d9ebeb0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:55.933880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3486e3c6-4fd1-4612-b57a-d70f0d9ebeb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:55.951842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3486e3c6-4fd1-4612-b57a-d70f0d9ebeb0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:55.953527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3486e3c6-4fd1-4612-b57a-d70f0d9ebeb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:55.997848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a6bbd9-6bd6-4ed9-9c33-e12cdd4a9cf8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.001674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:56.055444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:56.056699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:56.057215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:56.057679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:56.079164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:56.097875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:56.098809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:56.099591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:56.102385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 271/546] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 11:59:56.102599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 275/546] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:56.108457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3486e3c6-4fd1-4612-b57a-d70f0d9ebeb0 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:56.144817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:56.261369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:56.274828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aab988f-d5da-4c47-89ff-161bfaba998f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.297411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3486e3c6-4fd1-4612-b57a-d70f0d9ebeb0 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.311708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:56.312519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:56.312803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:56.313185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:56.324520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:56.340097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:56.341053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:56.341831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 276/547] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:56.349412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:56.360163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:56.361167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:56.361679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:56.362315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:56.363351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:56.363872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:56.365891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:56.378245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 272/548] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:56.383882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:56.429176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:56.534606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff239f92-d81e-4a5c-86ab-83526410be0a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.569854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e54698c8-dd48-49ba-bcee-596cf93b933f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.598077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:56.598749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:56.599143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:56.599414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:56.600274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:56.600582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:56.601962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:56.614759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 277/549] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:56.620342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:56.637078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:56.637877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:56.638153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:56.638430 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:56.647093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:56.663841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:56.665316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:56.691831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:56.692705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:56.693575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:56.693575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:56.693575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:56.693575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:56.701329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 273/550] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:56.708462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:56.796706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:56.822707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9853c3ed-7f79-4fbe-ab69-7ff4d2d16b5d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.844936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc40d560-f8e4-4b80-82a5-3a7c1ab694f0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.857836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:56.858553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:56.858923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:56.859207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:56.859939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:56.860282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:56.861648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:56.878858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 274/551] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:56.881211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:56.935493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:56.936308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:56.936589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:56.936864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:56.953207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:56.954681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:56.989007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:56.989925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:56.990806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:56.990806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:56.990806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:56.990806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:56.991024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d5f3ec-f274-46cc-9a6e-0c01e16aa9ee None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:56.994848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 278/552] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 370 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:56.996262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:57.026924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.027764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.028044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.028318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] PATH_INFO: `/v3/users/nova` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.030264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:57.038010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.039846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 11:59:57.061091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:57.062128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:57.063049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 275/553] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/users/nova => generated 82 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:57.063608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:57.065716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:57.107453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaf67ab-39df-4506-b3d7-261137c107fd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:57.146406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.147119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.147405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.147688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.148410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:57.148728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.159453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.168746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 279/554] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 11:59:57.177719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:57.222629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d86991-a6b2-40eb-a044-b55edc3e703a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:57.311339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7546b3a1-ad37-4ec2-8522-22a561b286bd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:57.324431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.325124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.325411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.325690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.326544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:57.326880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.328323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.335799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:57.349063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.349864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.350149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.354977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] PATH_INFO: `/v3/users/swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.358548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.360225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.369083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 276/555] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 11:59:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:57.374330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 11:59:57.376542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:57.377432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 14 11:59:57.380123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:57.383030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 280/556] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 11:59:57 2022] GET /identity/v3/users/swift => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:57.384211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:57.478317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:57.551673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e365c0-d02c-4166-8be9-8f3c043e87e5 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:57.551992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa2ff615-88c7-47c7-b811-e4b2e6289d5a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:57.591179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28555e57-cc27-4082-8e4c-add0c56f88f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.591897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28555e57-cc27-4082-8e4c-add0c56f88f1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.592178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28555e57-cc27-4082-8e4c-add0c56f88f1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.592452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28555e57-cc27-4082-8e4c-add0c56f88f1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.598313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359ab3bd-80a9-4d9c-acfe-922536b2e28c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.599027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ab3bd-80a9-4d9c-acfe-922536b2e28c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.599308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ab3bd-80a9-4d9c-acfe-922536b2e28c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.600066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359ab3bd-80a9-4d9c-acfe-922536b2e28c None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.600920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ab3bd-80a9-4d9c-acfe-922536b2e28c None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:57.601616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ab3bd-80a9-4d9c-acfe-922536b2e28c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.603379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28555e57-cc27-4082-8e4c-add0c56f88f1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.603580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359ab3bd-80a9-4d9c-acfe-922536b2e28c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.604829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28555e57-cc27-4082-8e4c-add0c56f88f1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.622989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 281/557] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Jul 14 11:59:57 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:57.624326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb008c1f-42fc-46cd-87fb-ee9a11ef5589 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.624609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb008c1f-42fc-46cd-87fb-ee9a11ef5589 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.624883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb008c1f-42fc-46cd-87fb-ee9a11ef5589 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.626379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 282/558] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-28555e57-cc27-4082-8e4c-add0c56f88f1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:57.632069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:57.633708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:57.635328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:57.635328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:57.635328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:57.635328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:57.635764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb008c1f-42fc-46cd-87fb-ee9a11ef5589 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:57.635915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 277/559] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 11:59:57.635915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457e287a-cfa9-46d2-ba60-ae0749df934e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.635915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457e287a-cfa9-46d2-ba60-ae0749df934e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.635915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457e287a-cfa9-46d2-ba60-ae0749df934e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.657165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-457e287a-cfa9-46d2-ba60-ae0749df934e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:57.658427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-457e287a-cfa9-46d2-ba60-ae0749df934e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:57.682613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:57.749635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb008c1f-42fc-46cd-87fb-ee9a11ef5589 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:57.758794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 278/560] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:57.758794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457e287a-cfa9-46d2-ba60-ae0749df934e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:57.790878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:57.809671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996fb104-0787-4706-a937-d83de96db12a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.810658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996fb104-0787-4706-a937-d83de96db12a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.811166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996fb104-0787-4706-a937-d83de96db12a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.811736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996fb104-0787-4706-a937-d83de96db12a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.879432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457e287a-cfa9-46d2-ba60-ae0749df934e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:57.887426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996fb104-0787-4706-a937-d83de96db12a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.889100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996fb104-0787-4706-a937-d83de96db12a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.923438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:57.924263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:57.925149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:57.925431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:57.927074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:57.927393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:57.928908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:57.933841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:57.946999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 279/561] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:57.946999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.000996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 283/562] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 11:59:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 11:59:58.003595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf9ed36-35d9-4973-8c67-b463addd2857 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.004099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf9ed36-35d9-4973-8c67-b463addd2857 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.004444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf9ed36-35d9-4973-8c67-b463addd2857 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.006035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 284/563] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:58.007562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf9ed36-35d9-4973-8c67-b463addd2857 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.049180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.056389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef0b1195-33b0-4a8b-89a5-9543f863c43c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.093206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.094128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.094414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.094692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.105974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:58.113918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:58.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:58.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 280/564] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:58.116447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.123460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf9ed36-35d9-4973-8c67-b463addd2857 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.158658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a52dfa9-a76e-46c9-a164-2c4899eb6b6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.159418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a52dfa9-a76e-46c9-a164-2c4899eb6b6f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.159697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a52dfa9-a76e-46c9-a164-2c4899eb6b6f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.159970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a52dfa9-a76e-46c9-a164-2c4899eb6b6f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.166554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a52dfa9-a76e-46c9-a164-2c4899eb6b6f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:58.168029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a52dfa9-a76e-46c9-a164-2c4899eb6b6f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:58.170816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-5a52dfa9-a76e-46c9-a164-2c4899eb6b6f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 11:59:58.182696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:58.184253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 285/565] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:58.185142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaccae3-ffc2-4c5c-85e5-8c637ad75645 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.185702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaccae3-ffc2-4c5c-85e5-8c637ad75645 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.185702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaccae3-ffc2-4c5c-85e5-8c637ad75645 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.201962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-acaccae3-ffc2-4c5c-85e5-8c637ad75645 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:58.203196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-acaccae3-ffc2-4c5c-85e5-8c637ad75645 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:58.223608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1ffdf1-25ac-4a62-a8d2-48c177925114 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.257667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.258452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.258730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.259027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.269461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:58.269698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 286/566] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:58.270932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:58.271082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b34dd5-ce8f-4ab2-a34f-1c6bcca56043 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.271318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b34dd5-ce8f-4ab2-a34f-1c6bcca56043 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.271596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9b34dd5-ce8f-4ab2-a34f-1c6bcca56043 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.272858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 287/567] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 11:59:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 11:59:58.273545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b34dd5-ce8f-4ab2-a34f-1c6bcca56043 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:58.298074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:58.299200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:58.300641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:58.300641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:58.300641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:58.300641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:58.300641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 281/568] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:58.301075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.310438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.383075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9b34dd5-ce8f-4ab2-a34f-1c6bcca56043 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.401733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0991e35-6fe7-40be-b48b-cfe78c2bafa0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.413141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.419930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3780e4e0-c4ac-4ea3-9547-18f36667849f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.420600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780e4e0-c4ac-4ea3-9547-18f36667849f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.420898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780e4e0-c4ac-4ea3-9547-18f36667849f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.421237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3780e4e0-c4ac-4ea3-9547-18f36667849f None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.421976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3780e4e0-c4ac-4ea3-9547-18f36667849f None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:58.422293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3780e4e0-c4ac-4ea3-9547-18f36667849f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:58.423708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3780e4e0-c4ac-4ea3-9547-18f36667849f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:58.432323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 288/569] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:58.433646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9a0f2-beff-4bfc-a3df-4d770414c569 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.433935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9a0f2-beff-4bfc-a3df-4d770414c569 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.434211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9a0f2-beff-4bfc-a3df-4d770414c569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.441440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.442104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.442390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.442669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.443493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:58.443806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:58.445258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:58.449852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bed9a0f2-beff-4bfc-a3df-4d770414c569 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 11:59:58.451058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bed9a0f2-beff-4bfc-a3df-4d770414c569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 11:59:58.459167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 282/570] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:58.459976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.516517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 289/571] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 11:59:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 11:59:58.517356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9a0f2-beff-4bfc-a3df-4d770414c569 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.536585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.572164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fe8dc9-bc24-4e5b-9d85-ff5c18a02c2b None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.608423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.609218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.609528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.609806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.620363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:58.621311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9a0f2-beff-4bfc-a3df-4d770414c569 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:58.628652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:58.630560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 283/572] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:58.631612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.648571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.661280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.664104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.664104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.664104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.664104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:58.664104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:58.665099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:58.677883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 290/573] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:58.678657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.740654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-545de4d7-cd1b-42bd-9053-818a817c1993 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.764950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.773185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.773963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.774497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.774799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.778904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af95175e-82eb-4f97-ae8d-a8420713bca2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.786185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:58.787655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:58.805122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:58.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:58.807415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:58.807415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:58.807415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:58.807415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:58.807415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 284/574] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:58.807415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.814593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.815556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.815838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.816109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/da24a7ecdcc249f0820add0404a7856c/roles/0fd45f27e71d455da928a71546fbe85e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.830129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=da24a7ecdcc249f0820add0404a7856c, role_id=0fd45f27e71d455da928a71546fbe85e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:58.831961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:58.904649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b00050-780d-4833-9579-b264dbcc90f7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:58.916121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 291/575] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 11:59:58 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/da24a7ecdcc249f0820add0404a7856c/roles/0fd45f27e71d455da928a71546fbe85e => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 11:59:58.916876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:58.941074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:58.941889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:58.942177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:58.942457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:58.953551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:58.955366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 11:59:58.964222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:58.972632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 285/576] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:58.973631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.016967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.041051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020e512a-fe29-4b06-af3f-e3f4438d70e6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.082615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.083345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.083634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.083907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.084656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:59.084974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.086349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.093196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13b8bad-2cf5-410f-a8f3-40c574ebde67 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.096150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 292/577] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:59.097116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.131576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.132343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.132620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.132889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.140070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.141546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 11:59:59.157038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:59.157945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 286/578] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 11:59:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:59.159429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.206125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.213475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-109a3cf8-3687-4439-b76a-596193c85a37 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.251050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.252046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.252328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.252793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.253627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:59.254294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.255335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.263290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.266864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991be550-d3b1-428e-aa2f-ca1a66763d61 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.273606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 293/579] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:59.273606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.311230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.311897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.312174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.312443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.313179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:59.313492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.314864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.327626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 287/580] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:59.327626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.385089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8181134e-ee1a-423a-b51e-5ee2771eb0c2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.424636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.425444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.425982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.426099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.434717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379e3a14-b58c-465e-b301-a3b6b18ee24a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.436744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.438198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:59.457261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:59.458327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:59.464024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:59.464024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:59.464024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:59.464024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:59.464024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 294/581] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:59.464024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.471849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.478527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.479460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.479803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.481849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.481849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:59.481849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.482961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.501270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 288/582] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:59.501270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.523436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.573695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35c6650-b6ee-494f-bf9e-b5b63f07884e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.607652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e899334c-3c0b-47c6-8275-eab0aa0c48e1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.613111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.613908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.614196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.614476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] PATH_INFO: `/v3/users/glance` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.622641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.624130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 11:59:59.639475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:59.640672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 14 11:59:59.641822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 11:59:59.645507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 295/583] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/users/glance => generated 84 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:59.646291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.647757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.648856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.649534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.650084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.662579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.664161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 11:59:59.682934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 11:59:59.683990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 11:59:59.685070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 11:59:59.685070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 11:59:59.685070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 11:59:59.685070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 11:59:59.685714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 289/584] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 11:59:59.686487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.724600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.761606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5649f3ba-6616-4fa4-a9f1-0b99da7e583f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.774604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.800892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.801545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.801824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.802105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.802843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:59.803150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.804150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec1ed75d-a850-465a-9146-5547a5687890 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.804527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.818630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 296/585] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:59.819484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.850389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.851339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.851621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.851895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.853180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=7b1634a6e6ec4db9a8851b1329ffe6a0) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:59.853622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.855272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 11:59:59.870802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 290/586] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/role_assignments?user.id=7b1634a6e6ec4db9a8851b1329ffe6a0&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 11:59:59.870802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 11:59:59.929929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1258ea54-088d-41a6-9386-22531b3e29df None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 11:59:59.972289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 11:59:59.988582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 11:59:59.988582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 11:59:59.988582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 11:59:59.988582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 11:59:59.988979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 11:59:59.988979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 11:59:59.990069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.012914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821f9cae-d38d-434d-a59b-08995aa3c957 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.014614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 297/587] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:00.015755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.075923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.076637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.076921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.077196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.078469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:00.078798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.080496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.100488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:00.104321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 291/588] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 11:59:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:00.105340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.141909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0440e7-59b1-4215-96dc-a2e4e4ee6f80 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.188230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.189032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.189313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.189583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.199013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.201033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.219241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d4c1c6-d1ee-449a-9281-979dc6a291bc None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:00.219727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:00.220665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 14 12:00:00.223112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:00.223635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 298/589] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/users/neutron => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:00.224186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.266667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.268780 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.269273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.270555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.283173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.283444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:00.285743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:00.307445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:00.308369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:00.309275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:00.309275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:00.309275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:00.309275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:00.311434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 292/590] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 206 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:00.314390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.357467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bc084f7-1a67-4766-aeec-bdd7e7d48235 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.368833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:00.402365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.403200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.403531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.403810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.413446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.414917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.429471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3b036e-98c4-495e-a2c2-d0936d1b8be8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:00.434565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:00.435511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:00.436427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 299/591] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/users/cinder => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:00.436999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.479173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.479842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.480122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.480396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.482016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:00.482332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.483796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.507476 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 293/592] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:00.508644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.559001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cdc87b-9d45-4b6b-b56b-1da4ec3c1a73 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.561311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:00.611144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.612923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.613447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.613991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.614730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:00.615077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.617167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.633736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 300/593] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:00.634946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.639509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a86929-3206-42e2-8274-696460a4aaa6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.649599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:00.680725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.682257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.682907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.683386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.685317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:00.690063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.692008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.716229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 294/594] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:00.717464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.771917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1094c09f-7d78-429b-988e-0f35c2c70d62 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.808818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.810116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.810607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.811127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.825247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.827131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.843838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4eb2685-2df3-463d-9932-6ec0cc54ef87 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:00.847895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:00.848756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:00.849645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:00.849645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:00.849645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:00.849645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:00.851918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 301/595] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:00.853010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.873008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:00.887181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:00.888159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:00.888442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:00.888720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:00.898176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:00.900342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:00.918235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:00.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 295/596] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:00.920294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:00.957222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:00.972800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a041c3-d71f-4716-a70f-1ea532cf0520 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.008737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.009706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.010081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.010460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.023272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.025152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.041198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58289c00-2a57-4aa4-89bf-cf2984709304 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:01.044322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:01.045187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:01.051807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:01.051807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:01.051807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:01.051807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:01.051807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 302/597] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:01.051807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.079182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fade5f-2402-4196-a482-96780b43fd5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.079771 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fade5f-2402-4196-a482-96780b43fd5d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.080056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fade5f-2402-4196-a482-96780b43fd5d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.080362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fade5f-2402-4196-a482-96780b43fd5d None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.081693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fade5f-2402-4196-a482-96780b43fd5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:01.081935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fade5f-2402-4196-a482-96780b43fd5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.083337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fade5f-2402-4196-a482-96780b43fd5d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.099427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 296/598] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:01.099427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f476a5ad-a674-4f7f-b369-4eeafac9171c None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.099427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f476a5ad-a674-4f7f-b369-4eeafac9171c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.099427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f476a5ad-a674-4f7f-b369-4eeafac9171c None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.099427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 297/599] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:01.099427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f476a5ad-a674-4f7f-b369-4eeafac9171c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.129568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:01.163512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9cf6ba-073d-4404-a1ff-e2739e0a9e8c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.196188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:01.200342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.201002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.201282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.201554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.202341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:01.202645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.204283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.215491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f476a5ad-a674-4f7f-b369-4eeafac9171c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.216558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 303/600] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:01.222239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.259300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267d4999-18aa-4339-a0d0-96e8e8765d73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.260093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d4999-18aa-4339-a0d0-96e8e8765d73 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.260509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d4999-18aa-4339-a0d0-96e8e8765d73 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.260959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267d4999-18aa-4339-a0d0-96e8e8765d73 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.261892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d4999-18aa-4339-a0d0-96e8e8765d73 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:01.262305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d4999-18aa-4339-a0d0-96e8e8765d73 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.264069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267d4999-18aa-4339-a0d0-96e8e8765d73 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.277152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 298/601] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:01.282819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed1fbc2-cbf5-49c7-835b-b5fb1a704211 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.283344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed1fbc2-cbf5-49c7-835b-b5fb1a704211 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.284269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed1fbc2-cbf5-49c7-835b-b5fb1a704211 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.301343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2ed1fbc2-cbf5-49c7-835b-b5fb1a704211 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:01.302915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2ed1fbc2-cbf5-49c7-835b-b5fb1a704211 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:01.316639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571f2c39-85c8-4138-a1ca-5374f8686b52 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.344852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:01.353317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.354188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.354538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.354963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.361714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.363484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.379905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 299/602] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:01.379905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed1fbc2-cbf5-49c7-835b-b5fb1a704211 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:01.380155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:01.381157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 304/603] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:01.382217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.463446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:01.489809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed1fbc2-cbf5-49c7-835b-b5fb1a704211 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.493627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d8a21c-def5-4101-822d-c18e620f74c3 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.526288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.527433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.527716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.527993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/86ace7e877e3484b8853536b77c76885/roles/0fd45f27e71d455da928a71546fbe85e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.530100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.530920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.531207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.532140 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.539106 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.540572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.543869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=86ace7e877e3484b8853536b77c76885, role_id=0fd45f27e71d455da928a71546fbe85e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.545370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:01.555295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:01.556320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:01.557262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 305/604] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:01.557741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.608686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:01.659591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 300/605] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:01 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/86ace7e877e3484b8853536b77c76885/roles/0fd45f27e71d455da928a71546fbe85e => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:00:01.659874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90449929-4146-4b83-b5d2-90e20686d4a2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.661668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.703875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.705075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.706558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.707008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.708351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:01.708820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.713972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.731576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 306/606] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:01.736328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.809867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed25b7a-eee7-4154-8fc1-226e120af66a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.842905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:01.856904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.857714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.858275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.858559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.873891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:01.884333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cdee64a-451e-41e8-8add-fd5a5f6a0e52 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:01.884622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:01.885533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:01.886354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 301/607] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:01.887174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:01.907846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:01.940423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:01.941475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:01.941912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:01.942339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:01.943302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:01.943848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:01.945593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:01.959580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 307/608] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:01.960830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.013402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60978fd6-625a-46c5-8f2d-ea8d2830e636 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.054567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59baa481-2ff8-4e31-a80c-2d9fce4424ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.055766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59baa481-2ff8-4e31-a80c-2d9fce4424ee None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.056063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59baa481-2ff8-4e31-a80c-2d9fce4424ee None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.056346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59baa481-2ff8-4e31-a80c-2d9fce4424ee None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/a4fff90ae5434b58932f468ca9c827f3/roles/0fd45f27e71d455da928a71546fbe85e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.068348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510ca07d-fcc8-44b0-90c3-f21519f3480f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.071548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59baa481-2ff8-4e31-a80c-2d9fce4424ee None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=a4fff90ae5434b58932f468ca9c827f3, role_id=0fd45f27e71d455da928a71546fbe85e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:02.072999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59baa481-2ff8-4e31-a80c-2d9fce4424ee None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:02.103751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.104416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.104696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.105022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.105696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:02.105997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:02.107405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:02.115868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 308/609] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:02.116451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.148799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:02.179558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 302/610] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:01 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/a4fff90ae5434b58932f468ca9c827f3/roles/0fd45f27e71d455da928a71546fbe85e => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:00:02.181301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43028c41-8240-42ce-a442-97a08ffff2ef None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.181830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43028c41-8240-42ce-a442-97a08ffff2ef None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.182319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43028c41-8240-42ce-a442-97a08ffff2ef None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.186562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 303/611] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:02.186562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43028c41-8240-42ce-a442-97a08ffff2ef None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.232388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:02.251801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18d02cb-59e7-40b8-a732-43ec406c273f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.300271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fd251-0701-4700-af0c-d193840e467d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.302008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fd251-0701-4700-af0c-d193840e467d None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.302700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fd251-0701-4700-af0c-d193840e467d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.303182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728fd251-0701-4700-af0c-d193840e467d None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/e114b292a7f544b0b24016199edcef25/roles/0fd45f27e71d455da928a71546fbe85e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.318267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43028c41-8240-42ce-a442-97a08ffff2ef None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.326175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728fd251-0701-4700-af0c-d193840e467d None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=e114b292a7f544b0b24016199edcef25, role_id=0fd45f27e71d455da928a71546fbe85e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:02.328585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728fd251-0701-4700-af0c-d193840e467d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:02.359445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-003f98b3-2a59-4a3d-b768-f3420e85dc75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.360452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003f98b3-2a59-4a3d-b768-f3420e85dc75 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.360841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003f98b3-2a59-4a3d-b768-f3420e85dc75 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.361223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003f98b3-2a59-4a3d-b768-f3420e85dc75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.372249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003f98b3-2a59-4a3d-b768-f3420e85dc75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:02.373809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-003f98b3-2a59-4a3d-b768-f3420e85dc75 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-003f98b3-2a59-4a3d-b768-f3420e85dc75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:02.392612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:02.394024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 304/612] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd17937-f8bc-4793-bfb4-b8d579d18be9 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd17937-f8bc-4793-bfb4-b8d579d18be9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.399986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecd17937-f8bc-4793-bfb4-b8d579d18be9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.413406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ecd17937-f8bc-4793-bfb4-b8d579d18be9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:02.414571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ecd17937-f8bc-4793-bfb4-b8d579d18be9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:02.455337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 309/613] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:02 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/e114b292a7f544b0b24016199edcef25/roles/0fd45f27e71d455da928a71546fbe85e => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:00:02.456777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fd251-0701-4700-af0c-d193840e467d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.509557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 305/614] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:02.533195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd17937-f8bc-4793-bfb4-b8d579d18be9 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.552543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:02.575452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:02.603998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728fd251-0701-4700-af0c-d193840e467d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.657395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecd17937-f8bc-4793-bfb4-b8d579d18be9 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.657836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.659647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.660084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.661482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.662432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:02.663010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:02.664569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:02.686438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 310/615] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:02.688506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.703208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.704005 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.704283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.704797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.717004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:02.725585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:02.726575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:02.727538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 306/616] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:02.732838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.809845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ea4e57-808c-43b6-a2d4-127c8376c820 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.818486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:02.841431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c036b08-1235-4191-86f4-b8aa82d1eed6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:02.848583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.849657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.850119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.850546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.859291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:02.860968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:02.864890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:02.878054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:02.882016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:02.882016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:02.882016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:02.882016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:02.882292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:02.883224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:02.884173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:02.884706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:02.884939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 311/617] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 12:00:02 2022] GET /identity/v3/users/nova => generated 82 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:02.885031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:02.889291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:02.895640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 307/618] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:02.901268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:03.006947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5cc8567-b8ac-4ce5-a864-e09360a62336 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:03.040462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3150fedd-13c4-4649-9f3e-b377434bfd9f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:03.047263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:03.048460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:03.048951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:03.049636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:03.051038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:03.051659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:03.053328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:03.074193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 312/619] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 12:00:02 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:03.080450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:03.083626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:03.084580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:03.084580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:03.084853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:03.094044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:03.106471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:03.108017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:03.135835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:03.136785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:03.138549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:03.138549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:03.138549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:03.138549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:03.138549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 308/620] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:03.147664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:03.156496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:03.203723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c576f3d9-f607-41e6-bbac-0df7855b68db None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:03.242238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:03.243269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:03.243853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:03.244334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:03.318727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:03.321221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:03.383464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:03.408819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f46fb43-27fe-4384-ae12-59b0190f7852 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:03.431813 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 313/621] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 14 12:00:03.436808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:03.474283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:03.475016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:03.475302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:03.475579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:03.476580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:03.476878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:03.479237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:03.498274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 309/622] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:03.503457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:03.588737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:03.601055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05af67e-8d70-4316-952b-77b2b8bb90c3 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:03.667873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:03.669152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:03.670419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:03.671571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:03.687966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b12d67-e208-4ce6-b0d6-9eef506a51cb None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:03.692176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:03.694120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:03.733269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:03.734328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:03.735462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:03.735462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:03.735462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:03.735462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:03.740188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 314/623] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 303 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:03.743965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:03.755729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:03.756868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:03.757316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:03.757738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] PATH_INFO: `/v3/users/swift` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:03.758312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:03.779887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:03.781584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:03.814810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:03.815976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:03.817094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:03.819113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 310/624] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:03 2022] GET /identity/v3/users/swift => generated 83 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:03.824565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:03.877787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887634d5-b60c-440d-9059-ffa779657670 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:03.919188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:03.919188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:03.919188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:03.921330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:03.921817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:03.922134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:03.923535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:03.951630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 315/625] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:03.956766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.045095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3660ea-7e73-4a54-bef5-931fd835bc5c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.052252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.072856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.090475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f9de481-67ea-4236-af10-cf0a45bec687 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.130935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c26e3517-0fc5-4c5b-ad45-60c2d451afc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.132160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e3517-0fc5-4c5b-ad45-60c2d451afc3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.132678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e3517-0fc5-4c5b-ad45-60c2d451afc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.133171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c26e3517-0fc5-4c5b-ad45-60c2d451afc3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.140880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e3517-0fc5-4c5b-ad45-60c2d451afc3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:04.142731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c26e3517-0fc5-4c5b-ad45-60c2d451afc3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:04.152595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1e728eb-e5ca-4cd9-8609-dd1f1f4ac4cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.153263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e728eb-e5ca-4cd9-8609-dd1f1f4ac4cd None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.153545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e728eb-e5ca-4cd9-8609-dd1f1f4ac4cd None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.153829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1e728eb-e5ca-4cd9-8609-dd1f1f4ac4cd None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.154658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e728eb-e5ca-4cd9-8609-dd1f1f4ac4cd None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:04.154991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e728eb-e5ca-4cd9-8609-dd1f1f4ac4cd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:04.156408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1e728eb-e5ca-4cd9-8609-dd1f1f4ac4cd None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-c26e3517-0fc5-4c5b-ad45-60c2d451afc3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:04.164023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:04.164915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:04.165882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:04.169486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 316/626] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:04.171106 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad85989-7cad-4dad-97c2-a43a7effb73c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.171669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad85989-7cad-4dad-97c2-a43a7effb73c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.172157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad85989-7cad-4dad-97c2-a43a7effb73c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.174004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 317/627] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:04.176532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad85989-7cad-4dad-97c2-a43a7effb73c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.183106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 311/628] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Jul 14 12:00:03 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 359 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:04.184389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e39e127-0f64-4013-b4d1-c8074733a345 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.184675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e39e127-0f64-4013-b4d1-c8074733a345 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.184952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e39e127-0f64-4013-b4d1-c8074733a345 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.208025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9e39e127-0f64-4013-b4d1-c8074733a345 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:04.209380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9e39e127-0f64-4013-b4d1-c8074733a345 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:04.285081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 312/629] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:04.286063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e39e127-0f64-4013-b4d1-c8074733a345 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.294005 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad85989-7cad-4dad-97c2-a43a7effb73c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.326317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.334698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.335705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.336211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.336657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.342507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:04.342507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:04.342507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:04.353986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 318/630] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:04.355362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.386276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.389736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e39e127-0f64-4013-b4d1-c8074733a345 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.426061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.426843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.427133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.427408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.437845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:04.439406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:04.457876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:04.458827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:04.460821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:04.460821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:04.460821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:04.460821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:04.460821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 313/631] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:04.460821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.475513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67792259-2b30-4227-9bd2-8217b3ddcd7f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.511872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab2d572-1324-4adf-bd41-4c0ba820035a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.512699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab2d572-1324-4adf-bd41-4c0ba820035a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.512981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab2d572-1324-4adf-bd41-4c0ba820035a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.513257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab2d572-1324-4adf-bd41-4c0ba820035a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.523980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ab2d572-1324-4adf-bd41-4c0ba820035a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-5ab2d572-1324-4adf-bd41-4c0ba820035a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:04.531884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:04.533014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:04.534639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:04.538418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 319/632] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/roles/service => generated 85 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:04.539658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac8533-b438-4918-9f85-d2ad82900de2 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.539950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac8533-b438-4918-9f85-d2ad82900de2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.540234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dac8533-b438-4918-9f85-d2ad82900de2 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.541555 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 320/633] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:04.542247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dac8533-b438-4918-9f85-d2ad82900de2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.573709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83732bac-a9aa-4f8c-8870-020b4e1fa128 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.593762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.611258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.612059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.612442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.612821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.613761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=da24a7ecdcc249f0820add0404a7856c) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:04.614185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:04.615751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:04.622020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 314/634] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/role_assignments?user.id=da24a7ecdcc249f0820add0404a7856c&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:04.623375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.654168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.657105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dac8533-b438-4918-9f85-d2ad82900de2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.712560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc6ae31-4a93-4d94-8093-cf6d89152b5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.713662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6ae31-4a93-4d94-8093-cf6d89152b5f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.715118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6ae31-4a93-4d94-8093-cf6d89152b5f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.716016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc6ae31-4a93-4d94-8093-cf6d89152b5f None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.717222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc6ae31-4a93-4d94-8093-cf6d89152b5f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:04.719119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc6ae31-4a93-4d94-8093-cf6d89152b5f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:04.722942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc6ae31-4a93-4d94-8093-cf6d89152b5f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:04.746031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 321/635] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:04.747638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce77193-f356-41a4-a15c-dc980928cb83 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.748063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce77193-f356-41a4-a15c-dc980928cb83 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.748500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bce77193-f356-41a4-a15c-dc980928cb83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.770982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ec52d89-9e3f-422d-810e-1073bb516238 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:04.772204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bce77193-f356-41a4-a15c-dc980928cb83 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:04.775117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bce77193-f356-41a4-a15c-dc980928cb83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:04.811261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f415a463-0c57-4755-8996-3261d066601a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:04.811261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f415a463-0c57-4755-8996-3261d066601a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:04.811261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f415a463-0c57-4755-8996-3261d066601a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:04.811701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f415a463-0c57-4755-8996-3261d066601a None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:04.812576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f415a463-0c57-4755-8996-3261d066601a None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:04.813030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f415a463-0c57-4755-8996-3261d066601a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:04.814497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f415a463-0c57-4755-8996-3261d066601a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:04.824363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 315/636] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:04.825296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f415a463-0c57-4755-8996-3261d066601a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.866069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 322/637] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:04.867061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce77193-f356-41a4-a15c-dc980928cb83 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:04.922921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.936091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:04.960827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f415a463-0c57-4755-8996-3261d066601a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.003242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.004522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.005073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.005642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.007312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bce77193-f356-41a4-a15c-dc980928cb83 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.014234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.016072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:05.032393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:05.033413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:05.034583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:05.042595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 316/638] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:05.043774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.047440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.048968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.048968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.048968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.060579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.062045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:05.080828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:05.081925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:05.082847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:05.082847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:05.082847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:05.082847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:05.082847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 323/639] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:05.087494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.160706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca42df7-9835-4b8a-8517-2d797cc956f2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.185308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:05.194022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405e11ef-c64f-4161-b5bb-4af104252d48 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.195587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.203804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.203804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.203804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.203804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:05.208502 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:05.216227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:05.217211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:05.218067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 317/640] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:05.218894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.235943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16d8e178-096a-4981-8d41-e822eeff6394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.236735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d8e178-096a-4981-8d41-e822eeff6394 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.237020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d8e178-096a-4981-8d41-e822eeff6394 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.237298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16d8e178-096a-4981-8d41-e822eeff6394 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.238269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d8e178-096a-4981-8d41-e822eeff6394 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:05.238587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d8e178-096a-4981-8d41-e822eeff6394 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.240089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16d8e178-096a-4981-8d41-e822eeff6394 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.254367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 324/641] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:05.256024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec6eba2-e02e-496a-af15-cb8d2f350b88 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.256313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec6eba2-e02e-496a-af15-cb8d2f350b88 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.256597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec6eba2-e02e-496a-af15-cb8d2f350b88 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.257819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 325/642] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:05.262354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec6eba2-e02e-496a-af15-cb8d2f350b88 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.326627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806c234a-84a7-4ba0-bc8a-d2bca6448cc4 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.363466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.364127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.364411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.364683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.365400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:05.365709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.367125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.369577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec6eba2-e02e-496a-af15-cb8d2f350b88 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.384483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 318/643] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:05.384483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.407159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c818dd2a-b7dc-4098-a545-d1b803e11708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.407826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c818dd2a-b7dc-4098-a545-d1b803e11708 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.408109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c818dd2a-b7dc-4098-a545-d1b803e11708 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.408405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c818dd2a-b7dc-4098-a545-d1b803e11708 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.409638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c818dd2a-b7dc-4098-a545-d1b803e11708 None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:05.410241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c818dd2a-b7dc-4098-a545-d1b803e11708 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.411348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c818dd2a-b7dc-4098-a545-d1b803e11708 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.419914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 326/644] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:05.421109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d022863-f1b3-4d95-9eb0-fdaff07699e1 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.421395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d022863-f1b3-4d95-9eb0-fdaff07699e1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.421673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d022863-f1b3-4d95-9eb0-fdaff07699e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.439764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-6d022863-f1b3-4d95-9eb0-fdaff07699e1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:05.441505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6d022863-f1b3-4d95-9eb0-fdaff07699e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:05.451392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:05.453108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:05.498999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b952a62a-09b8-4d2c-822a-f4b2274afaf2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.511476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 327/645] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:05.513435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d022863-f1b3-4d95-9eb0-fdaff07699e1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.540411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.541448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.541847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.542234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/7b1634a6e6ec4db9a8851b1329ffe6a0/roles/0fd45f27e71d455da928a71546fbe85e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.558600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=7b1634a6e6ec4db9a8851b1329ffe6a0, role_id=0fd45f27e71d455da928a71546fbe85e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.560208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.620087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d022863-f1b3-4d95-9eb0-fdaff07699e1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.647385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 319/646] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:05 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/7b1634a6e6ec4db9a8851b1329ffe6a0/roles/0fd45f27e71d455da928a71546fbe85e => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:00:05.648555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.664828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.666097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.666383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.666672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.677749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.679974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:05.701353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:05.702399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 14 12:00:05.703536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:05.706963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 328/647] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/users/glance => generated 84 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:05.708374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.715946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:05.786900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c375278-4574-4b89-aa3b-8f2f398e7161 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.795813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:05.828404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.829987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.831053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.831543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.836425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e39c74-cba2-47de-9afc-0735d362b732 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:05.846061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:05.847297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:05.870010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:05.871022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:05.871920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:05.871920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:05.871920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:05.871920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:05.871920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 320/648] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:05.872634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.881673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:05.882488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:05.882817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:05.883092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:05.895120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:05.903621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:05.904761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:05.905658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 329/649] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/roles/service => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:05.906345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:05.971875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:05.995255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a6c7f4-8bdb-4ad0-bb76-1bdbcbd6b337 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.010833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31131c64-bfab-44f3-81b6-2f457ec5ee42 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.033440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.034640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.035170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.035707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.036766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:06.037283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.039090 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.046009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.046668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.046972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.047244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.047967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:06.048271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.050185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.061548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 321/650] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:06.062800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.066988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 330/651] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:06.068403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.115193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:06.182488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f8e6749-24f6-4fec-9161-82834b90a7ae None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.191723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e56f29b-2ef6-4bda-b01a-1ad5b77844c2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.216804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:06.230160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.230160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.230160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.230160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.230666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.231340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.231611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.231881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.232623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:06.233306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.234787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.243875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 322/652] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:06.244764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.330967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.331168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:06.333233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.353846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b50bc1b9-c778-4f4d-ac39-bd8f065eb4f3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.389594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.390392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.390727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.390991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.400172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.401625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:06.416662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:06.417675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 14 12:00:06.418730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:06.422577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 323/653] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/users/neutron => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:06.423496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.472404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 331/654] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 12:00:06.474144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.508090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:06.532234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f69f484-c137-443a-a5fe-67b2eece8d55 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.568034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.568900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.569178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.569451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.576490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:06.582126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.583723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:06.601412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:06.602372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:06.603273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:06.603273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:06.603273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:06.603273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:06.603273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 324/655] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:06.604396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.611427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a915cba7-03be-4763-b182-d998b1258527 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.655627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.655989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.656366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.656736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.657673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:06.658086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.659755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.684008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 332/656] 10.222.0.27 () {56 vars in 1139 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:06.684891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.749937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec9ce02-bf7d-49c2-bf42-66ede3b89aa4 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.787575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.788710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.789218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.789764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.802465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.803484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:06.804616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:06.823603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:06.824544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:06.825403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:06.825403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:06.825403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:06.825403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:06.826433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1128dba-0076-4705-b881-e38dfbf7b0ab None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.828269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 325/657] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:06.829604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.880750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.882032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.882324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.882600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:06.883443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:06.884486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:06.885405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:06.886270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:06.900300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 333/658] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:06.901240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:06.958148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc04b116-43bf-4ac2-a9df-967a0f437856 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:06.996735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:06.997388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:06.997767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:06.998044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.025887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9ecd9-c9be-4efa-b479-19b734adfc0b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.064368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f4388c6-852d-4330-b3b0-49d6a50d805a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.065278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4388c6-852d-4330-b3b0-49d6a50d805a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.065650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4388c6-852d-4330-b3b0-49d6a50d805a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.065937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f4388c6-852d-4330-b3b0-49d6a50d805a None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.066662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4388c6-852d-4330-b3b0-49d6a50d805a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:07.067003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4388c6-852d-4330-b3b0-49d6a50d805a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.068372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f4388c6-852d-4330-b3b0-49d6a50d805a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.079001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.085201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 334/659] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:07.085355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.085454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce0179-b212-4531-b9b2-78f7a0d4755c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.085454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce0179-b212-4531-b9b2-78f7a0d4755c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.085454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ce0179-b212-4531-b9b2-78f7a0d4755c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.085454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 335/660] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:07.085454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce0179-b212-4531-b9b2-78f7a0d4755c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.088518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.135921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.199702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ce0179-b212-4531-b9b2-78f7a0d4755c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.209720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 326/661] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:06 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 12:00:07.209720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.246703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d801040-370f-4f30-8b43-f5acb5d4e1c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.247916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d801040-370f-4f30-8b43-f5acb5d4e1c3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.248727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d801040-370f-4f30-8b43-f5acb5d4e1c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.249236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d801040-370f-4f30-8b43-f5acb5d4e1c3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.262175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d801040-370f-4f30-8b43-f5acb5d4e1c3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.264190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d801040-370f-4f30-8b43-f5acb5d4e1c3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-9d801040-370f-4f30-8b43-f5acb5d4e1c3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:07.279451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:07.280654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 336/662] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/users/cinder => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:07.282076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a55d767-624d-482f-afda-681f7be28d67 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.282736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a55d767-624d-482f-afda-681f7be28d67 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.282736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a55d767-624d-482f-afda-681f7be28d67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.302355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0a55d767-624d-482f-afda-681f7be28d67 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:07.303627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0a55d767-624d-482f-afda-681f7be28d67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:07.320522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7321f3a3-2ea7-40ba-8d37-79bcdd1e93da None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.358331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.359154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.359434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.359813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.366801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.369438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.370969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.378340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 337/663] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:07.380297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a55d767-624d-482f-afda-681f7be28d67 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:07.385965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:07.387609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:07.389499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:07.392247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 327/664] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:07.393150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.396632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.505860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9552f661-3cd2-4dba-b4a3-093abe495bf3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.519227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a55d767-624d-482f-afda-681f7be28d67 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.543376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.547712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.547712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.547712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.547712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:07.547712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.547712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.556842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.557639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.557921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.558192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.569413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 328/665] 10.222.0.27 () {56 vars in 1137 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:07.570498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.574269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.575725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:07.594323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:07.595331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:07.596765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:07.596765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:07.596765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:07.596765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:07.596765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 338/666] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:07.597458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.639983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.645224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.683030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164c7a32-0a40-4814-a5c2-1b208e655d8c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.704560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e930612-d165-43b5-baa0-64184808f61a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.724941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.725612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.725895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.726168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.727238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:07.727315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.728737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.746066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 329/667] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:07.746066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.754593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.755526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.755897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.756244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.767484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:07.775467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:07.776580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:07.777560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:07.778203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 339/668] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:07.778495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.853774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c479bf-f58e-4916-8b40-f18f226cbd00 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.874706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.878781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d38530d-25fb-4ba8-bd95-4912b101edce None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:07.881076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:07.888334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.902527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.902527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.902527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.912660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:07.913326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:07.913610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:07.913913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:07.914682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:07.915039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.916443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:07.928432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 340/669] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:07.929244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:07.959987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:07.961452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.029880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f808cfc-418b-4a28-a6f5-305ca67680e8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.063339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61aad2d2-de79-44ca-9443-a577c4d453b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.063729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 330/670] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 12:00:08.063991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aad2d2-de79-44ca-9443-a577c4d453b8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.064271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aad2d2-de79-44ca-9443-a577c4d453b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.064545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61aad2d2-de79-44ca-9443-a577c4d453b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.065516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aad2d2-de79-44ca-9443-a577c4d453b8 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=86ace7e877e3484b8853536b77c76885) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:08.065823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aad2d2-de79-44ca-9443-a577c4d453b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.067308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61aad2d2-de79-44ca-9443-a577c4d453b8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.069032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.073359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 341/671] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:07 2022] GET /identity/v3/role_assignments?user.id=86ace7e877e3484b8853536b77c76885&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:08.074410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390291fa-ec33-4663-9c84-2f45a91f76f0 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.074696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390291fa-ec33-4663-9c84-2f45a91f76f0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.075004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390291fa-ec33-4663-9c84-2f45a91f76f0 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.076398 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 342/672] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:08 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:08.079639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390291fa-ec33-4663-9c84-2f45a91f76f0 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.093685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.122577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.188609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb9d63e8-6610-4aa8-a72f-5af020891426 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.208100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390291fa-ec33-4663-9c84-2f45a91f76f0 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.228222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.229470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.229470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.229470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.230211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:08.230536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.231964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.242653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 331/673] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:08.243168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.247494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f03091-a9e0-43bd-adcb-db888e347a45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.248359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f03091-a9e0-43bd-adcb-db888e347a45 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.249084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f03091-a9e0-43bd-adcb-db888e347a45 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.249084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f03091-a9e0-43bd-adcb-db888e347a45 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.258274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f03091-a9e0-43bd-adcb-db888e347a45 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.260087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f03091-a9e0-43bd-adcb-db888e347a45 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-64f03091-a9e0-43bd-adcb-db888e347a45 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:08.276142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:08.277117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:08.278022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 343/674] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:08.278985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0346e-809e-4e59-a9b6-1931750ec731 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.278985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0346e-809e-4e59-a9b6-1931750ec731 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.279154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76e0346e-809e-4e59-a9b6-1931750ec731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.297505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-76e0346e-809e-4e59-a9b6-1931750ec731 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:08.298719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-76e0346e-809e-4e59-a9b6-1931750ec731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:08.341320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.361361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfdbac6-c943-4a59-bc84-ca9859dc6698 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.378929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 344/675] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:08.378929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e0346e-809e-4e59-a9b6-1931750ec731 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.404294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.407928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.409366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.409651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.409927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.421058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.422504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:08.440748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:08.441865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:08.442875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:08.442875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:08.442875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:08.442875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:08.442875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 332/676] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:08.443184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.505953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76e0346e-809e-4e59-a9b6-1931750ec731 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.542262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.543242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.543534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.543820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.552092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baabe09-b743-47e9-9068-a692917d21fc None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.554948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.556493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:08.575356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:08.576301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:08.577200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:08.577200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:08.577200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:08.577200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:08.577200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 345/677] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:08.577986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.591081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.591829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.592182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.592534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.593428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:08.593842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.595500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.598197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.615525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 333/678] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:08.615525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.661389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.691121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1754d8-93f9-4683-b90d-f3a670977f1d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.717146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83731cb7-9b74-45bb-8615-8017d9bcfdaf None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.734324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.735251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.735528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.735798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.747549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:08.751672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.752995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.753365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.753649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.754369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:08.754678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:08.755858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:08.756736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:08.757557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:08.758178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.758247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 346/679] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:08.758330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.774743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 334/680] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:08.774743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.832340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.878589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c390cc-aea0-4c29-beb3-b0f5f760a114 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.880556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875aa040-6082-46b2-891e-81a225aec2b6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:08.909886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:08.920188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.920188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.920188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.920188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.920541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:08.920541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.920541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.920734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:08.921577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:08.922095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:08.922438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:08.923298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=a4fff90ae5434b58932f468ca9c827f3) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:08.923621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:08.925066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:08.936045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 347/681] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:08.936045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:08.939306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 335/682] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/role_assignments?user.id=a4fff90ae5434b58932f468ca9c827f3&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:08.940316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.069472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b48982f-d687-4278-bef8-a09d37d8a62a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.072978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd6d512-df88-42ab-9df2-bb6d9f3e9c9c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.088806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:09.108914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.109949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.110129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.110405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.118800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.120495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.125951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.126614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.127270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.127582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.128315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:09.128628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.130096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:09.136080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:09.136952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:09.137838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 336/683] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:09.138363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.139198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 348/684] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:08 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:09.144313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.208765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:09.262954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61077bae-7056-4958-9bb3-34e59cebc70c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.269911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9a8235b-f31e-44e8-b48e-f8c025537579 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.301620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.302356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.302646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.302956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.303762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:09.304242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.305800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.309573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.310370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.310651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.310956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.319716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 349/685] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:09.319877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.320631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.321427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:09.338182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:09.339189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:09.340129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 337/686] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:09.340716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.384995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:09.435244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ca594b4-d5e9-4ca1-96b2-e0f3f25e515c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.450023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56fc6ec5-1fe7-49f6-8e05-a34f4b503e3e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.451302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:09.475313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.476458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.476983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.477432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.488992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5142f59-268a-4714-a164-0803124da576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.489651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5142f59-268a-4714-a164-0803124da576 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.489932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5142f59-268a-4714-a164-0803124da576 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.490206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5142f59-268a-4714-a164-0803124da576 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.490463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.491077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5142f59-268a-4714-a164-0803124da576 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=e114b292a7f544b0b24016199edcef25) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:09.491387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5142f59-268a-4714-a164-0803124da576 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.492447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.492813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5142f59-268a-4714-a164-0803124da576 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.499480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 338/687] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/role_assignments?user.id=e114b292a7f544b0b24016199edcef25&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:09.500438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5142f59-268a-4714-a164-0803124da576 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:09.515541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:09.516688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:09.518314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:09.518314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:09.518314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:09.518314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:09.520540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 350/688] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:09.535105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.619460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5142f59-268a-4714-a164-0803124da576 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.621034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:09.657325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.658182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.658465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.658941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.660129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:09.660438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.666928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.682399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 339/689] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:09.688270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.737425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4399a31-7878-4227-8596-f71ffb4964b5 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.800822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1814b084-4617-4096-98da-8c415b3fcda1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:09.801112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.801963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.802395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.802851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.803745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:09.804193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.805709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.835434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 351/690] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:09.843623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:09.852136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:09.852898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:09.853244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:09.853586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:09.892731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:09.895847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:09.939293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:09.941790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:09.978337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cd9c50-4af7-483b-8e51-824468cdbce4 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.017024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.018633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.019160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.019639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.028446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.030365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:10.047989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:10.049092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:10.050781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:10.053458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 352/692] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/users/swift => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:10.057499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 340/692] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:09 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 14 12:00:10.059085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.059298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.133120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:10.171431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:10.184417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3f92f53-bdc1-4b89-b492-f9d7b9921646 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.188687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7d4e53-fd9c-47ba-8af7-84c64d97ba67 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.226302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.227536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.228055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.228548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.230134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.230810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.231231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.231670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.232907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:10.233233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.234717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.242524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.244403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.258488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 341/693] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:10.269899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:10.270831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:10.271743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:10.271743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:10.271743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:10.271743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:10.271971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.274637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 353/694] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:10.280037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.391900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49969b6d-6293-4f4c-ab00-db2719b81a1f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.410841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:10.428935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3b0d39-07d3-414e-bba0-842015ed81de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.436663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ffd988-b7af-4ed6-9af1-0435795fef33 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.436789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b0d39-07d3-414e-bba0-842015ed81de None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.436789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b0d39-07d3-414e-bba0-842015ed81de None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.436789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3b0d39-07d3-414e-bba0-842015ed81de None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.471915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee0dd44-3148-4678-bc71-5868db9779ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.472577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee0dd44-3148-4678-bc71-5868db9779ab None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.472857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee0dd44-3148-4678-bc71-5868db9779ab None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.473137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee0dd44-3148-4678-bc71-5868db9779ab None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.473971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee0dd44-3148-4678-bc71-5868db9779ab None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:10.474374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee0dd44-3148-4678-bc71-5868db9779ab None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.475796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee0dd44-3148-4678-bc71-5868db9779ab None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.478683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:10.493375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 354/695] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:10.494865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6477a483-2da8-4877-a865-54c262a92fb7 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.495161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6477a483-2da8-4877-a865-54c262a92fb7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.495468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6477a483-2da8-4877-a865-54c262a92fb7 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.497009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 355/696] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:10.497843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6477a483-2da8-4877-a865-54c262a92fb7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.519105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3b0d39-07d3-414e-bba0-842015ed81de None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.520818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3b0d39-07d3-414e-bba0-842015ed81de None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.628833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 342/697] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 14 12:00:10.628833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1122ada3-6582-4269-9a3c-21c6a0d8a602 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.628833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1122ada3-6582-4269-9a3c-21c6a0d8a602 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.628833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1122ada3-6582-4269-9a3c-21c6a0d8a602 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.631266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6477a483-2da8-4877-a865-54c262a92fb7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.645283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1122ada3-6582-4269-9a3c-21c6a0d8a602 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:10.646607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1122ada3-6582-4269-9a3c-21c6a0d8a602 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:10.663125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:10.668720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.669700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.670125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.670562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.678233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.679888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:10.696647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:10.697765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:10.698662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:10.701182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 356/698] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/projects/service => generated 88 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:10.702229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.726392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 343/699] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:10.727423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1122ada3-6582-4269-9a3c-21c6a0d8a602 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.769038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:10.820504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695d4cad-10b5-4467-bc17-3dc1de88cb71 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.859316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1122ada3-6582-4269-9a3c-21c6a0d8a602 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:10.860031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.861039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.862143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.862960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.875042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.876781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:10.895965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:10.896873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:10.897751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:10.897751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:10.897751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:10.897751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:10.899678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 357/700] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:10.900800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.905825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:10.906573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:10.906974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:10.907357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:10.908960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:10.909436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:10.911883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:10.927206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 344/701] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:10.928136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:10.944317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:11.034712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9829e483-95d5-472b-9c27-eee3d3856b8d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.044771 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ed7e010-300c-421f-bce0-2c9149ac89d6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.045168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:11.074457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.076402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.076687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.076966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.088901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.089963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.090251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.090529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.091566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:11.091892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.092711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.093291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.094862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.107022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 345/702] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:11.108315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:11.121191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:11.122128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:11.123697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:11.123697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:11.123697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:11.123697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:11.124102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 358/703] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:11.124894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:11.204046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:11.232396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f9580b-66d7-4120-b981-94f6909d24e9 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.249127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadffd58-0047-4c8b-a4fd-d5acfce960ff None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.268285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.268947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.269417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.269696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.270528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=da24a7ecdcc249f0820add0404a7856c) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:11.270868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.272317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.278178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 346/704] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:11 2022] GET /identity/v3/role_assignments?user.id=da24a7ecdcc249f0820add0404a7856c&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:11.279087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:11.294514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c69bd4-1bfa-4daf-a130-86705836f097 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.295628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c69bd4-1bfa-4daf-a130-86705836f097 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.296181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c69bd4-1bfa-4daf-a130-86705836f097 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.296617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c69bd4-1bfa-4daf-a130-86705836f097 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.304967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c69bd4-1bfa-4daf-a130-86705836f097 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.306847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c69bd4-1bfa-4daf-a130-86705836f097 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-d8c69bd4-1bfa-4daf-a130-86705836f097 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:11.323491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:11.324700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:11.325814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:11.333404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 359/705] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:11.335220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aad8cd-b608-4eb0-b8ed-4e5fb693ac8d None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.335660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aad8cd-b608-4eb0-b8ed-4e5fb693ac8d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.336125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6aad8cd-b608-4eb0-b8ed-4e5fb693ac8d None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.337774 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 360/706] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:11.338787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aad8cd-b608-4eb0-b8ed-4e5fb693ac8d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:11.364084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:11.412492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27b884dd-69bb-4da5-b180-2fe372f77ca4 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.456801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6aad8cd-b608-4eb0-b8ed-4e5fb693ac8d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.458365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:11.466642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53bc9bc-9f81-42de-8219-74a7a54ac2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.466642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bc9bc-9f81-42de-8219-74a7a54ac2d2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.466642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bc9bc-9f81-42de-8219-74a7a54ac2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.478297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bc9bc-9f81-42de-8219-74a7a54ac2d2 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.478297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bc9bc-9f81-42de-8219-74a7a54ac2d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:11.478297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bc9bc-9f81-42de-8219-74a7a54ac2d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.478297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bc9bc-9f81-42de-8219-74a7a54ac2d2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.481077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 347/707] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:11.484094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a29c03f-bcc1-4111-b664-545f66496dfd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.484094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a29c03f-bcc1-4111-b664-545f66496dfd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.484094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a29c03f-bcc1-4111-b664-545f66496dfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.499441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-5a29c03f-bcc1-4111-b664-545f66496dfd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:11.500939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5a29c03f-bcc1-4111-b664-545f66496dfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:11.502992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.504067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.504376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.504710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.506514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:11.506857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.508383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.528429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 361/708] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:11.529444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:11.577904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 348/709] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:11.578910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a29c03f-bcc1-4111-b664-545f66496dfd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:11.621596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:11.641791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-495b24b0-00d4-4e7b-9469-39ef54dbeef4 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.679599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.680521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.680956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.681456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.682344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.684007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.688136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a29c03f-bcc1-4111-b664-545f66496dfd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.716653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:11.731294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.731959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.732241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.732518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.733349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=7b1634a6e6ec4db9a8851b1329ffe6a0) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:11.733665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.735120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.746937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 349/710] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:11 2022] GET /identity/v3/role_assignments?user.id=7b1634a6e6ec4db9a8851b1329ffe6a0&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 608 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:11.746937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:11.808151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 362/711] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 12:00:11 2022] POST /identity/v3/users => generated 254 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:11.896255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-566f9fe2-933c-4692-ba88-0fa87d98b0d7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:11.935046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:11.935926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:11.936214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:11.936495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] PATH_INFO: `/v3/services/network` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:11.937221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:11.938665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:11.954496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:11.955672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:11.956723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 350/712] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:11 2022] GET /identity/v3/services/network => generated 88 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:11.962615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:12.083692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1855f5-6720-4a16-950c-ca1074e0ff43 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:12.118652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64b1addc-1018-405a-ba67-322a407e3bbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:12.119354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1addc-1018-405a-ba67-322a407e3bbb None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:12.119654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1addc-1018-405a-ba67-322a407e3bbb None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:12.119920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64b1addc-1018-405a-ba67-322a407e3bbb None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:12.120687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1addc-1018-405a-ba67-322a407e3bbb None admin] RBAC: Adding query filter params (name=network) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:12.121065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1addc-1018-405a-ba67-322a407e3bbb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:12.122496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64b1addc-1018-405a-ba67-322a407e3bbb None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:12.124459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:12.129400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 363/713] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:11 2022] GET /identity/v3/services?name=network => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:12.135081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:12.192187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:12.209544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bc398-efde-4466-afa6-7cac57184576 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:12.209860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bc398-efde-4466-afa6-7cac57184576 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:12.210105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-268bc398-efde-4466-afa6-7cac57184576 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:12.211715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 364/714] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:12.216818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152ecb05-7e50-430b-8310-9153a246e863 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:12.217105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152ecb05-7e50-430b-8310-9153a246e863 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:12.217383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-152ecb05-7e50-430b-8310-9153a246e863 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:12.237602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-152ecb05-7e50-430b-8310-9153a246e863 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:12.239376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-152ecb05-7e50-430b-8310-9153a246e863 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:12.257668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ea064a-4ce1-42b0-92a8-1c91a9124886 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:12.297100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:12.298600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:12.299147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:12.299659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:12.300576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] RBAC: Adding query filter params (type=network) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:12.301170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:12.302785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:12.309645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 351/715] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:12 2022] GET /identity/v3/services?type=network => generated 124 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:12.329516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 365/716] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 12:00:12.352842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:12.497225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8f4b30-7c67-4805-8e3a-2d79e6bd8dd9 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:12.515979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:12.540676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:12.541499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:12.541782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:12.542055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:12.542725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:12.544209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:12.558243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:12.559264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:12.560183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 352/717] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Jul 14 12:00:12 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:12.567913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152ecb05-7e50-430b-8310-9153a246e863 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:12.685019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-152ecb05-7e50-430b-8310-9153a246e863 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:12.723159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:12.727816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7d35c0b-8de9-421d-96dd-4584772daa4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:12.728419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d35c0b-8de9-421d-96dd-4584772daa4c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:12.728703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d35c0b-8de9-421d-96dd-4584772daa4c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:12.729321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7d35c0b-8de9-421d-96dd-4584772daa4c None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:12.730035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d35c0b-8de9-421d-96dd-4584772daa4c None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:12.730349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d35c0b-8de9-421d-96dd-4584772daa4c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:12.731772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7d35c0b-8de9-421d-96dd-4584772daa4c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:12.736751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 366/718] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Jul 14 12:00:12 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:12.742251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:12.882713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e086e7e4-f9d4-4fd7-8c5c-a22d013001a3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:12.918012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b0df36-8321-4b25-b1a0-9d9b2d25bceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:12.918682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b0df36-8321-4b25-b1a0-9d9b2d25bceb None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:12.919071 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b0df36-8321-4b25-b1a0-9d9b2d25bceb None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:12.919350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b0df36-8321-4b25-b1a0-9d9b2d25bceb None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:12.920066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b0df36-8321-4b25-b1a0-9d9b2d25bceb None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:12.920375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b0df36-8321-4b25-b1a0-9d9b2d25bceb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:12.921741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b0df36-8321-4b25-b1a0-9d9b2d25bceb None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:12.927570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 353/719] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Jul 14 12:00:12 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:13.794150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b63077b-14b8-43b7-8c23-6c3eb76e82c0 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:13.794693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b63077b-14b8-43b7-8c23-6c3eb76e82c0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:13.794693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b63077b-14b8-43b7-8c23-6c3eb76e82c0 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:13.796337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 367/720] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:13.808138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1293ea-3c2c-416d-9b95-7117e8befeca None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:13.808564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1293ea-3c2c-416d-9b95-7117e8befeca None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:13.808762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa1293ea-3c2c-416d-9b95-7117e8befeca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:13.835728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-fa1293ea-3c2c-416d-9b95-7117e8befeca None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:13.836945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fa1293ea-3c2c-416d-9b95-7117e8befeca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:13.879865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:13.923617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 354/721] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:13 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:13.959598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b63077b-14b8-43b7-8c23-6c3eb76e82c0 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:14.071217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b63077b-14b8-43b7-8c23-6c3eb76e82c0 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:14.106002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:14.106818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.107104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.107385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.118480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:14.132128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:14.133436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:14.134311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 368/722] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:13 2022] GET /identity/v3/roles/admin => generated 83 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:14.139294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1293ea-3c2c-416d-9b95-7117e8befeca None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:14.280507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa1293ea-3c2c-416d-9b95-7117e8befeca None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:14.327582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:14.328177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.328462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.328743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.329564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:14.329880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:14.331590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:14.346837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 355/723] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:14.352428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:14.410454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:14.457379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc214da-6b9d-4191-8605-de0dce60e753 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:14.493541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e156b28-413d-435d-aaa0-bd92b8ee62ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:14.494362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e156b28-413d-435d-aaa0-bd92b8ee62ed None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.494647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e156b28-413d-435d-aaa0-bd92b8ee62ed None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.494950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e156b28-413d-435d-aaa0-bd92b8ee62ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.506012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e156b28-413d-435d-aaa0-bd92b8ee62ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:14.507506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e156b28-413d-435d-aaa0-bd92b8ee62ed None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-7e156b28-413d-435d-aaa0-bd92b8ee62ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:14.525035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:14.527189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:14.529650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:14.529650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:14.529650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:14.529650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:14.529650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 369/724] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:14 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:14.532726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:14.569409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:14.665164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a6d265-a16a-4050-95b5-abee9888de08 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:14.681891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8d9eec-0818-4260-b447-c59f285fa5f1 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.681891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8d9eec-0818-4260-b447-c59f285fa5f1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.681891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f8d9eec-0818-4260-b447-c59f285fa5f1 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.684158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 370/725] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:14.696257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068503cf-ce04-41af-96ce-de1b1d1bf8f0 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.696675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068503cf-ce04-41af-96ce-de1b1d1bf8f0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.697058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068503cf-ce04-41af-96ce-de1b1d1bf8f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.721602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:14.722991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.723418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.723824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.724691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:14.725139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:14.726668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:14.730844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-068503cf-ce04-41af-96ce-de1b1d1bf8f0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:14.730844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-068503cf-ce04-41af-96ce-de1b1d1bf8f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:14.753131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 356/726] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:14.757167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:14.826631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 371/727] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 14 12:00:14.836886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b9fe43-b1c9-4f63-9cbc-e4dad75fa896 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.837174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b9fe43-b1c9-4f63-9cbc-e4dad75fa896 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.837542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b9fe43-b1c9-4f63-9cbc-e4dad75fa896 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.840250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 372/728] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:14 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:14.842531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c0df6-8ccb-4452-8ab3-b3ef817b0b89 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.844051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c0df6-8ccb-4452-8ab3-b3ef817b0b89 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.844333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af7c0df6-8ccb-4452-8ab3-b3ef817b0b89 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.845753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 373/729] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:14.847873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae5a5ca-bf63-4a91-a109-d4865bcebe40 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.848387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae5a5ca-bf63-4a91-a109-d4865bcebe40 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.848860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae5a5ca-bf63-4a91-a109-d4865bcebe40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.867597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-aae5a5ca-bf63-4a91-a109-d4865bcebe40 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:14.868794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-aae5a5ca-bf63-4a91-a109-d4865bcebe40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:14.880258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:14.887970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae368ba-7229-4287-bde1-4cb0b4732433 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:14.901350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:14.928035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:14.928751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.929162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.929437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] PATH_INFO: `/v3/users/nova` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.936889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:14.938519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:14.942694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 374/730] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:14.943903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dfc425-f930-4786-a077-023894ae7fcf None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:14.944189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dfc425-f930-4786-a077-023894ae7fcf None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:14.944462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dfc425-f930-4786-a077-023894ae7fcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:14.960643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:14.961570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:14.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:14.963515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e2dfc425-f930-4786-a077-023894ae7fcf None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:14.963515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e2dfc425-f930-4786-a077-023894ae7fcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:14.965117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 357/731] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 12:00:14 2022] GET /identity/v3/users/nova => generated 82 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:14.967684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.033482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 375/732] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:15.034441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dfc425-f930-4786-a077-023894ae7fcf None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.090389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4af1c9-bca7-4d23-99fa-c4ba4c7e8e8f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.126261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.127451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.127451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.127649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.138341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:15.146524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:15.147652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 358/733] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:15.151598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.154971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:15.161884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dfc425-f930-4786-a077-023894ae7fcf None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.166787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:15.202789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.203475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.203757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.204030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.204748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:15.206308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:15.253473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a4c69ab-34e9-4e95-8cb2-9770174f582e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.287255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.287945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.288224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.288496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.289318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:15.289626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:15.291070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:15.309207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 359/734] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:15.309915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.321094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 376/735] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:15 2022] POST /identity/v3/services => generated 240 bytes in 287 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:15.322036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.396856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:15.435811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0b69954-1274-4fe3-82d1-fdfaae318a8d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.474973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced60619-cadf-414f-a17c-cab84455b421 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.478589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.479658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.480127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.480609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.498798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:15.501180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:15.502323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:15.503380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:15.508378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 360/736] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:15.508674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.534541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.535298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.535585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.535862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.536940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:15.536940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:15.538814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:15.549576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 377/737] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:15.550238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.573810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:15.664078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc91764-a9a9-4a1b-94ad-55119278d01c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.699459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:15.703971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cda4551-8e8e-4142-8a25-be05f028bd2d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.704828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.705795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.706127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.706354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.718438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:15.719937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:15.740179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:15.741089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:15.741954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:15.741954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:15.741954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:15.741954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:15.748636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 361/738] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 236 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:15.748636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.753691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e28bb0a-bf6d-4459-8e96-d0842cd071ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.754433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e28bb0a-bf6d-4459-8e96-d0842cd071ab None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.755209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e28bb0a-bf6d-4459-8e96-d0842cd071ab None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.755490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e28bb0a-bf6d-4459-8e96-d0842cd071ab None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.756249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e28bb0a-bf6d-4459-8e96-d0842cd071ab None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:15.756897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e28bb0a-bf6d-4459-8e96-d0842cd071ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:15.758683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e28bb0a-bf6d-4459-8e96-d0842cd071ab None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:15.767982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 378/739] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:15.769322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b6a51-384b-4f2c-8cc2-a9905a6dc430 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.769608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b6a51-384b-4f2c-8cc2-a9905a6dc430 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.769888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00b6a51-384b-4f2c-8cc2-a9905a6dc430 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.775335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 379/740] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:15.776118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b6a51-384b-4f2c-8cc2-a9905a6dc430 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.858148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:15.866625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b242901-2cec-4459-8986-6a7282a01633 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.883513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00b6a51-384b-4f2c-8cc2-a9905a6dc430 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:15.907477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816f9abb-a990-4612-a988-3347ff39c340 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.909726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816f9abb-a990-4612-a988-3347ff39c340 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.910326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816f9abb-a990-4612-a988-3347ff39c340 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.911021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816f9abb-a990-4612-a988-3347ff39c340 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.919039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:15.920086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.920379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.920656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.921666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:15.921813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816f9abb-a990-4612-a988-3347ff39c340 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:15.921984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:15.923260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816f9abb-a990-4612-a988-3347ff39c340 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:15.923375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:15.941737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 380/741] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:15.941737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-816f9abb-a990-4612-a988-3347ff39c340 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:15.942991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:15.943830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:15.945308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:15.945308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:15.945308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:15.945308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:15.945308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 362/742] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:15.945713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72088e53-6829-45ee-8afc-c54de1337ba0 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:15.945954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72088e53-6829-45ee-8afc-c54de1337ba0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:15.946230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72088e53-6829-45ee-8afc-c54de1337ba0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:15.963471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-72088e53-6829-45ee-8afc-c54de1337ba0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:15.964581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-72088e53-6829-45ee-8afc-c54de1337ba0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:15.994986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.038965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 363/743] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:15 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 14 12:00:16.047747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72088e53-6829-45ee-8afc-c54de1337ba0 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.051667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e6577e9-a282-48af-8bcb-d915304b1c91 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.087502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.088298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.088587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.088864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.091633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.099716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.101193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:16.119154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:16.120075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:16.120948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:16.120948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:16.120948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:16.120948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:16.124837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 381/744] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:16.125691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.160025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72088e53-6829-45ee-8afc-c54de1337ba0 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.197856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.198650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.199215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.199500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.208937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.210837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:16.228767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:16.229671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:16.230580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:16.231529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67392830-57c3-43f7-a6b5-668ccae70a2e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.231617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 364/745] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:16.231617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.260386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.271759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.272424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.272705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.272982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.273699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:16.274014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.275672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.287534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 382/746] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:16.292466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.338061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76f115c-8c6d-4c17-b762-1c12a630cf58 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.342301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.372980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.373644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.373922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.374196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.375105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.376427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.400853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37fe7b5-6367-49f3-9e3e-ad9edefb0c8d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.435028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.435681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.435959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.436236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.436951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:16.437256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.438622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.452596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 383/747] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:16.453466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.463546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 365/748] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:16 2022] POST /identity/v3/services => generated 251 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:16.464601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.573473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.602593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3123de19-f2eb-43e2-8701-5de83a66f6fd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.610519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6fa0bb-60d3-4c84-83fa-6b053fe45b91 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.614756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.644416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.645086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.645369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.646788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.646982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:16.647253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.648704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.653600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.654375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.654655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.655552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] PATH_INFO: `/v3/users/swift` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.664673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.666241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.670493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 384/749] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:16.670493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:16.687156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:16.688321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:16.690047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:16.693046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 366/750] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/users/swift => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:16.693764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.801714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2d7b57-e718-4b8c-b840-31202d4cd9bc None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.836816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.837585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.837868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.838142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.841769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed9e493-5a1d-48a5-bb5c-865c91084aba None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:16.847078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.848533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:16.864660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:16.866083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 385/751] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:16.867053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.889134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:16.890915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:16.891426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:16.892188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/da24a7ecdcc249f0820add0404a7856c/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:16.895201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=da24a7ecdcc249f0820add0404a7856c, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:16.897377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:16.897692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.964918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 367/752] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:16 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/da24a7ecdcc249f0820add0404a7856c/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:00:16.967280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:16.970529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:16.996232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cbd540e-efb5-4554-99fe-731907173945 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:17.076222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:17.078061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.078728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.080092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.081729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:17.082668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:17.085272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:17.113603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 386/753] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 247 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:17.119829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:17.185512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557dc173-ca3f-4c5c-b795-7a88ad630fc2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:17.227475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:17.228147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.228426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.228695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.229646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:17.229958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:17.231890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:17.248858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:17.254524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 368/754] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Jul 14 12:00:16 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:17.260293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:17.270318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a39915-c7f0-4375-8cba-58272f7ab199 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:17.314891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e72e0438-f0be-46b2-bdec-bf859a00d8d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:17.315573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72e0438-f0be-46b2-bdec-bf859a00d8d8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.315854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72e0438-f0be-46b2-bdec-bf859a00d8d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.316147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e72e0438-f0be-46b2-bdec-bf859a00d8d8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.327848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:17.372392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbd85e7f-4d4d-430b-aff5-4fdc6918ab6f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:17.413847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72e0438-f0be-46b2-bdec-bf859a00d8d8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:17.415386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e72e0438-f0be-46b2-bdec-bf859a00d8d8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:17.416675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44d02a62-3ced-4ad0-8527-8a9d07d65f48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:17.417607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d02a62-3ced-4ad0-8527-8a9d07d65f48 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.418168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d02a62-3ced-4ad0-8527-8a9d07d65f48 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.419044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d02a62-3ced-4ad0-8527-8a9d07d65f48 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.508627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:17.517405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d02a62-3ced-4ad0-8527-8a9d07d65f48 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:17.519791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44d02a62-3ced-4ad0-8527-8a9d07d65f48 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:17.525168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 387/755] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:17 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:00:17.526601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a298f4ef-f9ee-4f81-b92c-4066b89f8a8a None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.526931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a298f4ef-f9ee-4f81-b92c-4066b89f8a8a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.527211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a298f4ef-f9ee-4f81-b92c-4066b89f8a8a None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.528759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 388/756] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:17.530064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a298f4ef-f9ee-4f81-b92c-4066b89f8a8a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:17.597212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:17.646547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 369/757] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:17 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:00:17.647983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab8e76-3ef8-4377-ac90-6a6390c6aed2 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.648254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab8e76-3ef8-4377-ac90-6a6390c6aed2 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.648540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dab8e76-3ef8-4377-ac90-6a6390c6aed2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.664023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a298f4ef-f9ee-4f81-b92c-4066b89f8a8a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:17.666411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1dab8e76-3ef8-4377-ac90-6a6390c6aed2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:17.667633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1dab8e76-3ef8-4377-ac90-6a6390c6aed2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:17.710699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:17.711868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.712382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.712877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.729521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:17.731983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:17.751144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 370/758] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:17.752222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dab8e76-3ef8-4377-ac90-6a6390c6aed2 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:17.757216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:17.758272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:17.759396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:17.759396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:17.759396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:17.759396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:17.761257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 389/759] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:17.766170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:17.820381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:17.862101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:17.874916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dab8e76-3ef8-4377-ac90-6a6390c6aed2 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:17.910122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b34f97-2b94-4c6a-bf64-d1854a7d45c2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:17.913123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:17.914178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.914664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.915195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.927007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:17.928774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:17.948368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:17.949268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:17.950192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:17.950192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:17.950192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:17.950192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:17.956057 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 371/760] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:17.956917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:17.957240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:17.958133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:17.958613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:17.959111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:17.960079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:17.960654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:17.962235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:17.975822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 390/761] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:17.977045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.076835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce77e96-6975-4b95-9a7b-a54fa68435d7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.094659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:18.100012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f588a8c1-d90d-4b1b-a76e-a1cc69832fcb None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.115361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.116447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.116934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.117442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] PATH_INFO: `/v3/services/network` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.119469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.121473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:18.133886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:18.134885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:18.135706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:18.138684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 372/762] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:17 2022] GET /identity/v3/services/network => generated 88 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:18.139775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.146576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.147529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.148036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.148506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.149674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:18.150204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.151797 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.154943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:18.172133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 391/763] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:18.172133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.265900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88fca6d-5540-486a-8bba-4ac4b1951252 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.296615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-691aadb5-99e3-40b3-aadc-bc2a42307c9e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.305995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.307559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.308071 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.308959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.318687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.321310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.334519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.335207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.335484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.335758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.336462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] RBAC: Adding query filter params (name=network) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:18.336765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.338203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:18.342947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:18.343909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:18.344782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:18.348189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 373/765] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:18.348421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 392/765] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/services?name=network => generated 124 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:18.349513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.353627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.381595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:18.423073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:18.462356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2324a59d-8045-4dbe-b6e1-183e8dbc76a6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.472156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0a5fec-5104-4641-80b4-de3c9d29ed1d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.499431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.500099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.500386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.500664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.501461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:18.501778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.503348 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.515244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.516377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.516872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.517358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.520834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 374/766] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:18.521697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.526598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.528347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:18.544455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:18.546960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:18.547890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:18.554061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 393/767] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:18.555673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.626410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840662c5-3094-49f7-b7e8-e20b68f538ed None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.648998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:18.653123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:18.662193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.662889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.663212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.663566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.664224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] RBAC: Adding query filter params (type=network) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:18.664534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.665922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.676370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 375/768] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/services?type=network => generated 350 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:18.676370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.676600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f67b9e-a705-4d5d-9a32-8a7a30d85e97 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.720238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.722174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.723069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.723550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.724601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=7b1634a6e6ec4db9a8851b1329ffe6a0) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:18.725214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.726978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.739136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 394/769] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/role_assignments?user.id=7b1634a6e6ec4db9a8851b1329ffe6a0&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:18.740289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.836907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b193a578-d754-415c-80c7-ea8bd0914491 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.864531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847a49a9-6531-47b5-b997-f0341a84f90c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:18.884567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df2e48c-36d8-430a-b306-2299d10bc270 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.885238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df2e48c-36d8-430a-b306-2299d10bc270 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.885521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df2e48c-36d8-430a-b306-2299d10bc270 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.885798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df2e48c-36d8-430a-b306-2299d10bc270 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.886610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df2e48c-36d8-430a-b306-2299d10bc270 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:18.886957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df2e48c-36d8-430a-b306-2299d10bc270 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.888656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df2e48c-36d8-430a-b306-2299d10bc270 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.907385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 376/770] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:18.907385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ce7b0-75d0-48ab-93d0-0f3204d3e954 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.907385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ce7b0-75d0-48ab-93d0-0f3204d3e954 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.907385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84ce7b0-75d0-48ab-93d0-0f3204d3e954 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.907385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 377/771] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:18.913526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ed6a4-0509-482d-91d4-083a36933e46 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.913819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ed6a4-0509-482d-91d4-083a36933e46 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.914103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-646ed6a4-0509-482d-91d4-083a36933e46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.917082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:18.918282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:18.918726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:18.919501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:18.921524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] RBAC: Adding query filter params (interface=public, service_id=4cc3a30214a246709b3c6fff0f47d8e7, region_id=RegionOne) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:18.921988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:18.923566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:18.937494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-646ed6a4-0509-482d-91d4-083a36933e46 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:18.938642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-646ed6a4-0509-482d-91d4-083a36933e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:18.939331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 395/772] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/endpoints?service_id=4cc3a30214a246709b3c6fff0f47d8e7&interface=public®ion_id=RegionOne => generated 194 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:18.940532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:18.953804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:19.004284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:19.039233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 378/773] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:18 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:19.039814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646ed6a4-0509-482d-91d4-083a36933e46 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:19.062350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b426d9b-1b51-4155-9944-3bca0829645d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:19.100343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:19.101056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.101339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.101612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.102662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=1843ddb84e8b434dbed5ebd32419b8d6) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:19.102878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:19.104368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:19.118968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 396/774] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:18 2022] GET /identity/v3/role_assignments?user.id=1843ddb84e8b434dbed5ebd32419b8d6&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:19.121295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:19.186829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-646ed6a4-0509-482d-91d4-083a36933e46 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:19.231936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:19.257302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:19.258229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.258513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.258812 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.259518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:19.260940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:19.274920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 379/775] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:19 2022] GET /identity/v3/services => generated 772 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:19.279054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50ca2ed-6bd7-4ea9-9de3-cde0899f1611 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:19.319612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-337304d0-0cf7-4c97-906f-8ef36a0691f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:19.320456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337304d0-0cf7-4c97-906f-8ef36a0691f7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.320739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337304d0-0cf7-4c97-906f-8ef36a0691f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.321014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337304d0-0cf7-4c97-906f-8ef36a0691f7 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.321703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337304d0-0cf7-4c97-906f-8ef36a0691f7 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:19.323217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-337304d0-0cf7-4c97-906f-8ef36a0691f7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-337304d0-0cf7-4c97-906f-8ef36a0691f7 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:19.339443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:19.340450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:19.341292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 397/776] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Jul 14 12:00:19 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:19.347506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:19.353529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc6489b-877b-4a45-81b1-bdd6117e407f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.353873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc6489b-877b-4a45-81b1-bdd6117e407f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.354163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc6489b-877b-4a45-81b1-bdd6117e407f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.355861 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 398/777] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:19.364596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77902b5e-0406-41f6-9a05-cb44b83738e9 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.364925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77902b5e-0406-41f6-9a05-cb44b83738e9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.365151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77902b5e-0406-41f6-9a05-cb44b83738e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.384480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-77902b5e-0406-41f6-9a05-cb44b83738e9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:19.385733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-77902b5e-0406-41f6-9a05-cb44b83738e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:19.407276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:19.485070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4e0b1-67a1-4366-b46c-5a0129083338 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:19.506884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 399/778] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:19 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:19.532911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:19.535454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77902b5e-0406-41f6-9a05-cb44b83738e9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:19.535611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.535832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.536121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.536891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:19.537208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:19.538847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:19.544478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 380/779] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Jul 14 12:00:19 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:19.553513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:19.677845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:19.692954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf68aeb5-cb97-4e6e-972d-8bdc9179dad1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:19.708706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:19.737008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77902b5e-0406-41f6-9a05-cb44b83738e9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:19.760448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:19.761128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.761413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.761688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.762397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:19.762706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:19.764126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:19.777186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 381/780] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Jul 14 12:00:19 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:19.783626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:19.804200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:19.804998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:19.805284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:19.805555 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:19.828874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:19.838775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:19.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:19.840566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:19.841925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 400/781] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:19.855304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:19.971167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db443deb-b54b-44aa-92fb-8cf7b6dabfb2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:19.995895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1c1eb3-6bd0-48c9-b565-c928e012187b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:20.024114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.025054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.025339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.025620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] PATH_INFO: `/v3/endpoints` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:20.025921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:20.026485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] RBAC: Adding query filter params (interface=public, service_id=b182506eff7d4c7ebc6932ac1c33faff, region_id=RegionOne) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:20.026739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:20.028152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:20.034929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.035602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.035881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.036157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:20.036909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:20.037221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:20.038661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:20.046672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 382/782] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:00:19 2022] GET /identity/v3/endpoints?service_id=b182506eff7d4c7ebc6932ac1c33faff&interface=public®ion_id=RegionOne => generated 194 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:20.048261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 401/783] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:20.054451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:20.054665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:20.115222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:20.166436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef6028ed-767d-4864-98ba-209330fb13f8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:20.203580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.204831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.206063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.206617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:20.218013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:20.219768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:20.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:20.240746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:20.241643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:20.241643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:20.241643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:20.241643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:20.243893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b05c6fe-7b15-4f28-8297-07e2c880100c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:20.244444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 402/784] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:20.249992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:20.297688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:20.311466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.312158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.312454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.312743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:20.313448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:20.315402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:20.321186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 383/785] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:20 2022] GET /identity/v3/services => generated 772 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:20.362557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f10615d-df28-4089-952b-5ebba5ec5f77 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:20.400966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.402078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.402595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.403103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:20.404060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:20.404582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:20.406229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:20.420704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 403/786] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:20.427157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:20.538562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:20.543792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81232f17-d47a-4e1f-91a8-e1747914bff2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:20.590050 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.590871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.591157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.591436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:20.599771 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:20.601282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:20.616044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:20.617028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:20.618397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 384/787] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 12:00:20 2022] GET /identity/v3/users/nova => generated 82 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:20.623212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:20.735821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b561eb92-6477-43c8-9ab9-2195295715ce None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:20.738288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:20.773095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.774154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.775187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.775460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:20.776297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:20.776601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:20.778188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:20.796665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 404/788] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 12:00:20 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:20.803047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:20.925356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76cc2383-2e2a-46ec-874b-eb2949563e74 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:20.983271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e937ccf-7fb5-4251-80af-b39ab3538b2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:20.984487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e937ccf-7fb5-4251-80af-b39ab3538b2b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:20.984992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e937ccf-7fb5-4251-80af-b39ab3538b2b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:20.985491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e937ccf-7fb5-4251-80af-b39ab3538b2b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:21.042253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:21.112355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e937ccf-7fb5-4251-80af-b39ab3538b2b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:21.115024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e937ccf-7fb5-4251-80af-b39ab3538b2b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:21.315755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 385/789] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:20 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 513 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:00:21.324467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:21.424199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87c8ee-a703-41b2-ad41-64b2f0b94809 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:21.424646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87c8ee-a703-41b2-ad41-64b2f0b94809 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:21.425144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87c8ee-a703-41b2-ad41-64b2f0b94809 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:21.427679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 386/790] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:21 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:21.432760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4b62b8-4fd7-4724-98ab-acc2fa20e34c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:21.433076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4b62b8-4fd7-4724-98ab-acc2fa20e34c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:21.433343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4b62b8-4fd7-4724-98ab-acc2fa20e34c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:21.471863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2a4b62b8-4fd7-4724-98ab-acc2fa20e34c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:21.473074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2a4b62b8-4fd7-4724-98ab-acc2fa20e34c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:21.514732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef572e0f-c8c5-4b35-b7b2-0967b28fc079 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:21.580918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:21.583017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:21.583799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:21.584451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:21.607531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:21.610597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:21.639241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:21.649163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:21.651164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:21.653492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:21.653492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:21.653492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:21.653492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:21.653492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 405/791] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:21.661716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:21.677135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 387/792] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:21 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:21.714595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4b62b8-4fd7-4724-98ab-acc2fa20e34c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:21.719762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:21.863494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061d8114-0e8b-4c34-ad59-ffcfb847afc0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:21.931813 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:21.932505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:21.932793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:21.933072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:21.933814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:21.934126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:21.935551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:21.948877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 406/793] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:21.954695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.072664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4b62b8-4fd7-4724-98ab-acc2fa20e34c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.074340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:22.113531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ce7020-6585-4489-a119-5bae802a9831 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.117515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46752787-c238-4491-b742-85705116f94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.118338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46752787-c238-4491-b742-85705116f94b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.118608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46752787-c238-4491-b742-85705116f94b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.118920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46752787-c238-4491-b742-85705116f94b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.132269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46752787-c238-4491-b742-85705116f94b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-46752787-c238-4491-b742-85705116f94b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:22.142966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:22.144023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:22.145420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 388/794] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 434 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:22.146128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf7f2cc-ade2-4b0d-954a-87ef96a4df0d None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.146128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf7f2cc-ade2-4b0d-954a-87ef96a4df0d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.146459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf7f2cc-ade2-4b0d-954a-87ef96a4df0d None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.152452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 389/795] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:22.152452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf7f2cc-ade2-4b0d-954a-87ef96a4df0d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.162181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-338d08f8-75a1-4b69-b653-90e00e0a51d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.163023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338d08f8-75a1-4b69-b653-90e00e0a51d9 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.163585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338d08f8-75a1-4b69-b653-90e00e0a51d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.163859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-338d08f8-75a1-4b69-b653-90e00e0a51d9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.171211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338d08f8-75a1-4b69-b653-90e00e0a51d9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.172671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-338d08f8-75a1-4b69-b653-90e00e0a51d9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-338d08f8-75a1-4b69-b653-90e00e0a51d9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:22.187343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:22.188220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 407/796] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:21 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:22.189791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39fa96-859b-49c5-92f9-d299af6a70be None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.190355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39fa96-859b-49c5-92f9-d299af6a70be None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.190355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b39fa96-859b-49c5-92f9-d299af6a70be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.208407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0b39fa96-859b-49c5-92f9-d299af6a70be None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:22.209611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0b39fa96-859b-49c5-92f9-d299af6a70be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:22.213769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:22.267918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf7f2cc-ade2-4b0d-954a-87ef96a4df0d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.290965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 408/797] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:22.290965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7743f4-cd47-4536-8152-018c8c4c077f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.290965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7743f4-cd47-4536-8152-018c8c4c077f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.290965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7743f4-cd47-4536-8152-018c8c4c077f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.290965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 409/798] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:22.290965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7743f4-cd47-4536-8152-018c8c4c077f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.314195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadee87a-c834-421f-a163-bd0d42ccca17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.314904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadee87a-c834-421f-a163-bd0d42ccca17 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.315187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadee87a-c834-421f-a163-bd0d42ccca17 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.315464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadee87a-c834-421f-a163-bd0d42ccca17 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.316214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadee87a-c834-421f-a163-bd0d42ccca17 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:22.316788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadee87a-c834-421f-a163-bd0d42ccca17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.317890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cadee87a-c834-421f-a163-bd0d42ccca17 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.327112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 390/799] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:22.328323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aa9f3e-cf7d-4f51-80f0-e2258468df90 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.328605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aa9f3e-cf7d-4f51-80f0-e2258468df90 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.328965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aa9f3e-cf7d-4f51-80f0-e2258468df90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.347496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-97aa9f3e-cf7d-4f51-80f0-e2258468df90 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:22.348684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-97aa9f3e-cf7d-4f51-80f0-e2258468df90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:22.370585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:22.414187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7743f4-cd47-4536-8152-018c8c4c077f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.417415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 391/800] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:22 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:22.424473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97aa9f3e-cf7d-4f51-80f0-e2258468df90 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.451937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.452750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.453306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.453520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.454457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:22.455139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.456685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.473335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 410/801] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:22.473335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.484691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:22.543937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97aa9f3e-cf7d-4f51-80f0-e2258468df90 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.579211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.580872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.581142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3daa1df3-7c39-44dc-b497-d916cd9123a5 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.581453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.581890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.593009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:22.600862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:22.602020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:22.603042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:22.604070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 392/802] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:22.609193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.609306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:22.620967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.622014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.622447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.622964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.636168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.638729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:22.660916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:22.662078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:22.663381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:22.663381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:22.663381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:22.663381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:22.668973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 411/803] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:22.668973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.730409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af39cc8-9bc5-4cd3-80fa-b4748bc37d4a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.732126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:22.767525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.768099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.768378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.768652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] PATH_INFO: `/v3/services/network` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.769308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.769891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea030dd-4eda-4c8c-8680-389344023277 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.770663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:22.782520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:22.784231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:22.785253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 393/804] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/services/network => generated 88 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:22.786226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.806720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.807405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.807703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.807970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.808799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=da24a7ecdcc249f0820add0404a7856c) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:22.809118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.810559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.816843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 412/805] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/role_assignments?user.id=da24a7ecdcc249f0820add0404a7856c&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:22.817608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.869471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:22.896643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7f691d-1fe2-4933-b817-5f1fdb477933 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.933631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7b4586-c96c-4398-bf8c-ab27a261f004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.934447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b4586-c96c-4398-bf8c-ab27a261f004 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.934834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b4586-c96c-4398-bf8c-ab27a261f004 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.935186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7b4586-c96c-4398-bf8c-ab27a261f004 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.935958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7b4586-c96c-4398-bf8c-ab27a261f004 None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:22.936276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7b4586-c96c-4398-bf8c-ab27a261f004 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.937686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7b4586-c96c-4398-bf8c-ab27a261f004 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.941123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c589a7-8500-4c6b-a0fe-b4984faf884e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:22.952117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 394/806] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:22.952117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65cb94d-894f-43e0-9b36-578b7236f43b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.952117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65cb94d-894f-43e0-9b36-578b7236f43b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.952117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65cb94d-894f-43e0-9b36-578b7236f43b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.955036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 395/807] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:22.955970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65cb94d-894f-43e0-9b36-578b7236f43b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:22.984102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:22.985071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:22.985360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:22.985636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:22.987107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:22.987583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:22.988998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:22.991328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.006817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 413/808] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:23.006817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.085218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65cb94d-894f-43e0-9b36-578b7236f43b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.114899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.118747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0753ee24-9e3c-4de7-9618-2bfa3b85048d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.122189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c536601-a5bc-4b47-b16e-5e93e4bb306c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.122886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c536601-a5bc-4b47-b16e-5e93e4bb306c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.128716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c536601-a5bc-4b47-b16e-5e93e4bb306c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.128716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c536601-a5bc-4b47-b16e-5e93e4bb306c None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.128716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c536601-a5bc-4b47-b16e-5e93e4bb306c None admin] RBAC: Adding query filter params (name=network) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:23.128716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c536601-a5bc-4b47-b16e-5e93e4bb306c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.128716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c536601-a5bc-4b47-b16e-5e93e4bb306c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.131601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 396/809] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:22 2022] GET /identity/v3/services?name=network => generated 124 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:23.132757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef977b69-5651-4ebf-9816-08f97256b46f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.133046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef977b69-5651-4ebf-9816-08f97256b46f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.133377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef977b69-5651-4ebf-9816-08f97256b46f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.151013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ef977b69-5651-4ebf-9816-08f97256b46f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:23.152180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ef977b69-5651-4ebf-9816-08f97256b46f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:23.160506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6675de17-869b-4605-b520-85cc6584674c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.161011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675de17-869b-4605-b520-85cc6584674c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.161299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675de17-869b-4605-b520-85cc6584674c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.161577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6675de17-869b-4605-b520-85cc6584674c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.172474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6675de17-869b-4605-b520-85cc6584674c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.174402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6675de17-869b-4605-b520-85cc6584674c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-6675de17-869b-4605-b520-85cc6584674c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:23.192923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:23.193868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:23.195757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:23.195757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:23.195757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:23.195757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:23.195757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 414/810] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:23.195757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6675de17-869b-4605-b520-85cc6584674c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.229138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 397/811] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:23 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:23.230315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef977b69-5651-4ebf-9816-08f97256b46f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.261547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.316928 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6675de17-869b-4605-b520-85cc6584674c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.342981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef977b69-5651-4ebf-9816-08f97256b46f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.352466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.353267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.353549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.353822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.361359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.362850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:23.378359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:23.379779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:23.380718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:23.381250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.381250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.381250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.381250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.381511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] RBAC: Adding query filter params (type=network) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:23.381511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.382179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.384405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 415/812] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/users/swift => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:23.385260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.391846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 398/813] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/services?type=network => generated 350 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:23.392889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.398852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.483084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.495494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9bb3d15-bc5a-4a34-a9c5-ce0e5a2e21b4 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.498226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff1736f-a158-43c8-a437-d94f9e4d4bbe None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.530966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.531547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.531830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.532352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.533668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:23.533965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.534066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.534803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.535092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.539922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.540042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.540042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.540042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.552376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 416/814] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:23.552376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:23.552642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:23.553488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:23.554335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 399/815] 10.222.0.27 () {56 vars in 1129 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:23.555223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.627265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.665939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4537dc-c705-4aca-905c-96d67ab2ef42 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.677940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a05fea7-259c-4f5c-942d-a5ceb2168a17 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.700042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.700722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.701004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.701289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.702126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:23.702438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.703909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.719573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.720534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.721004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.721463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 417/816] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:23.721757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] PATH_INFO: `/v3/endpoints` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.722268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.723019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.724573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.822738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f70a6af-a4c1-4dd4-aedb-3a1cc0ae8e06 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:23.830867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 400/817] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:23 2022] POST /identity/v3/endpoints => generated 327 bytes in 272 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:23.830867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.838874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.873181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:23.889746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:23.891093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:23.891442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:23.891728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:23.903127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:23.905312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:23.927654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:23.928687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 418/818] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:23.934953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:23.967568 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4952354-7798-4340-9066-82cb43df7e9b None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.010523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.011363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.011744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.012901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.013722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:24.014130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.015631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.020815 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 401/819] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:24.021693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.065435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f2ccd6-7d88-4d49-9784-424c1cd8d74c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.102627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.103723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.104155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.104701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.117275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.118213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.119832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:24.138485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:24.139524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:24.140405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:24.140405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:24.140405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:24.140405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:24.140405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 419/820] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:24.141097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.156461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cf52e6-3c81-45ef-b13c-e0bced37ba08 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.180106 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.192814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.193488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.193773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.194102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.194972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:24.195293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.196852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.215711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 402/821] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:24.216513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.254741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9815f2-a12d-4e59-b06b-3c74ca97ca81 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.292336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.292999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.293283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.293561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.294268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:24.294579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.296110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.307127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 420/822] 10.222.0.27 () {56 vars in 1124 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:24.307127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.324661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217dfc8a-fba8-483e-8029-53a4f86912bc None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.353087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.362655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.372433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.372433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.372433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.372433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:24.372433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.372433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.383601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 403/823] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:24.383601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.408671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.411562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66cb2d90-144c-4d19-820e-8c39af30a703 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.448534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.449637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.449926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.450199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.460777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.462229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:24.480286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:24.481213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:24.482130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:24.482130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:24.482130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:24.482130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:24.482130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 421/824] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:24.483371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.497026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c29ed4-959e-4a2a-b411-bd77a7a267d3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.532801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.533598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.533879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.534161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.534945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.536314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.590938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a173a4-80ec-4fca-94b5-7ad4f090e2b7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.626298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.627599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.627877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.628164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.634966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.636423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.647072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 404/825] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:24 2022] POST /identity/v3/endpoints => generated 346 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:24.648029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:24.651258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:24.652171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:24.658256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 422/826] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:24.661215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.682510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.718781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.787491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ee1d86-28eb-4ef2-9cad-df58c5dd1801 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.793435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df334a7b-8539-443b-9ceb-9f4d84308166 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.827135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.828120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.828613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.829223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.830244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:24.830846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.832248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:24.832738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.833087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:24.833438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:24.833782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:24.834579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:24.834984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:24.836549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:24.853891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 405/828] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:24.854075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 423/828] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:24.855581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.857795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:24.959035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.975172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:24.978020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad2213-daa1-4b3f-82b8-fd1706ce6d1a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:24.991386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-398d83a9-9df4-4d6e-9638-abbec1775bb6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:25.014271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:25.015260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.015541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.015813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/7b1634a6e6ec4db9a8851b1329ffe6a0/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.018688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=7b1634a6e6ec4db9a8851b1329ffe6a0, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:25.021348 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:25.028943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e520c9f5-c8e0-47a2-8c52-89875b252789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:25.029765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e520c9f5-c8e0-47a2-8c52-89875b252789 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.030048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e520c9f5-c8e0-47a2-8c52-89875b252789 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.030328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e520c9f5-c8e0-47a2-8c52-89875b252789 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.037409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e520c9f5-c8e0-47a2-8c52-89875b252789 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:25.038862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e520c9f5-c8e0-47a2-8c52-89875b252789 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-e520c9f5-c8e0-47a2-8c52-89875b252789 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:25.052802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:25.054852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 424/829] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b0e44-9d3d-484c-935f-572c0dc12450 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.055821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b0e44-9d3d-484c-935f-572c0dc12450 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.056400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-155b0e44-9d3d-484c-935f-572c0dc12450 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.057173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 425/830] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:25 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:25.062232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a245fd61-1eae-437e-b656-a95b95875a2c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.062517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a245fd61-1eae-437e-b656-a95b95875a2c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.062827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a245fd61-1eae-437e-b656-a95b95875a2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.083123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 406/831] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:24 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/7b1634a6e6ec4db9a8851b1329ffe6a0/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:00:25.083123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:25.084032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a245fd61-1eae-437e-b656-a95b95875a2c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:25.085290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a245fd61-1eae-437e-b656-a95b95875a2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:25.162693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 426/832] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:25 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:25.185583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a245fd61-1eae-437e-b656-a95b95875a2c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:25.225015 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8492415-33c3-4035-b810-65c521d2579e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:25.261793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:25.276266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:25.277076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.277462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.277834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.278730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:25.279186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:25.280730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:25.296438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 407/833] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:25.302802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:25.318221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a245fd61-1eae-437e-b656-a95b95875a2c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:25.330195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:25.364308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:25.365534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.366072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.366352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.367236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:25.369213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:25.386234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:25.387492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:25.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 427/834] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Jul 14 12:00:25 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:25.394147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:25.425465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2dc7b51-786c-4cc3-a55b-41ce8bbe1e50 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:25.472140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:25.473414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.473908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.474389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/1843ddb84e8b434dbed5ebd32419b8d6/roles/0fd45f27e71d455da928a71546fbe85e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.494467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=1843ddb84e8b434dbed5ebd32419b8d6, role_id=0fd45f27e71d455da928a71546fbe85e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:25.496336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:25.504913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567f4afc-9546-44ac-9418-e8889db1f8a2 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:25.533597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:25.543020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc07a1b-9f6a-4f43-ad29-e56b440c8d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:25.543284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 408/835] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:25 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/1843ddb84e8b434dbed5ebd32419b8d6/roles/0fd45f27e71d455da928a71546fbe85e => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:00:25.543723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc07a1b-9f6a-4f43-ad29-e56b440c8d57 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.543996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc07a1b-9f6a-4f43-ad29-e56b440c8d57 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.544272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc07a1b-9f6a-4f43-ad29-e56b440c8d57 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.544986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc07a1b-9f6a-4f43-ad29-e56b440c8d57 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:25.545297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc07a1b-9f6a-4f43-ad29-e56b440c8d57 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:25.546673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc07a1b-9f6a-4f43-ad29-e56b440c8d57 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:25.557058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 428/836] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:25 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:25.563317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:25.644946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:25.678550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94622584-b7ab-4016-a846-eedef8baa2c0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:25.721422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5db930-06c9-49fc-9d06-da4227d64c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:25.722281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5db930-06c9-49fc-9d06-da4227d64c95 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:25.722666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5db930-06c9-49fc-9d06-da4227d64c95 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:25.723098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5db930-06c9-49fc-9d06-da4227d64c95 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:25.723920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5db930-06c9-49fc-9d06-da4227d64c95 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:25.724333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5db930-06c9-49fc-9d06-da4227d64c95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:25.726069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e5db930-06c9-49fc-9d06-da4227d64c95 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:25.733867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 409/837] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:25 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:26.834225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936298ba-0572-4bb2-9314-f6e09c6a1828 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:26.834793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936298ba-0572-4bb2-9314-f6e09c6a1828 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:26.834793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-936298ba-0572-4bb2-9314-f6e09c6a1828 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:26.836510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 429/838] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:26.841583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0d5ad-c0ad-433c-b9fa-8b203c8431e3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:26.841989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0d5ad-c0ad-433c-b9fa-8b203c8431e3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:26.842178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f0d5ad-c0ad-433c-b9fa-8b203c8431e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:26.859786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f6f0d5ad-c0ad-433c-b9fa-8b203c8431e3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:26.861119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f6f0d5ad-c0ad-433c-b9fa-8b203c8431e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:26.935779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 410/839] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:26 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:26.957657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936298ba-0572-4bb2-9314-f6e09c6a1828 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.181105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-936298ba-0572-4bb2-9314-f6e09c6a1828 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:27.216218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10e8af3-5489-4a2b-8134-77cfe4405c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:27.217013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e8af3-5489-4a2b-8134-77cfe4405c8d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.217287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e8af3-5489-4a2b-8134-77cfe4405c8d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.217564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10e8af3-5489-4a2b-8134-77cfe4405c8d None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.218213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10e8af3-5489-4a2b-8134-77cfe4405c8d None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:27.219596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10e8af3-5489-4a2b-8134-77cfe4405c8d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-a10e8af3-5489-4a2b-8134-77cfe4405c8d None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:27.231857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:27.234169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:27.236269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 430/840] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:26 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:27.241460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f0d5ad-c0ad-433c-b9fa-8b203c8431e3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.264026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba79d9-7fe8-4622-a6f3-3a22335a6eee None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.264395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba79d9-7fe8-4622-a6f3-3a22335a6eee None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.264748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bba79d9-7fe8-4622-a6f3-3a22335a6eee None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.266411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 431/841] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:27.272100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f8879-fb94-477c-acb5-73be44a00282 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.272396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f8879-fb94-477c-acb5-73be44a00282 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.272679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f8879-fb94-477c-acb5-73be44a00282 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.289911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4a5f8879-fb94-477c-acb5-73be44a00282 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:27.291175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4a5f8879-fb94-477c-acb5-73be44a00282 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:27.323200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:27.342564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:27.360499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 432/842] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:27 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:27.360674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f0d5ad-c0ad-433c-b9fa-8b203c8431e3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:27.386100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f8879-fb94-477c-acb5-73be44a00282 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.407494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:27.408699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.409217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.409722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.411317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:27.412118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:27.414368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:27.421487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 411/843] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:27 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:27.427880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.506403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f8879-fb94-477c-acb5-73be44a00282 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:27.544469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:27.545332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.545616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.545961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.556327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49791a08-9bbb-4ca6-8cb8-d5b15d924b18 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:27.557967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:27.566578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:27.567576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:27.568424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:27.572871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 433/844] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:27.573732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.590591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:27.597948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e7a137c-bbff-49a2-aebc-8de063629ffe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:27.598731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a137c-bbff-49a2-aebc-8de063629ffe None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.599277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a137c-bbff-49a2-aebc-8de063629ffe None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.599653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e7a137c-bbff-49a2-aebc-8de063629ffe None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.602725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7a137c-bbff-49a2-aebc-8de063629ffe None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:27.602725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7a137c-bbff-49a2-aebc-8de063629ffe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:27.602997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e7a137c-bbff-49a2-aebc-8de063629ffe None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:27.610181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 412/845] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:27 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:27.630441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:27.697976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00163179-74f5-49ed-9688-a808667d8e01 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:27.702152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc2f124-c83a-46b7-85dc-1ddf0b1fce62 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.702152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc2f124-c83a-46b7-85dc-1ddf0b1fce62 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.702401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc2f124-c83a-46b7-85dc-1ddf0b1fce62 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.703936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 413/846] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:27.716042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01febf7-f4e5-425e-9893-21a23ddd7f82 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.716341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01febf7-f4e5-425e-9893-21a23ddd7f82 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.716610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d01febf7-f4e5-425e-9893-21a23ddd7f82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.740641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d01febf7-f4e5-425e-9893-21a23ddd7f82 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:27.741893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d01febf7-f4e5-425e-9893-21a23ddd7f82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:27.744191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:27.745187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.745469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.745799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.746579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:27.746933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:27.748400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:27.758630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 434/847] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:27.764075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.831502 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 414/848] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:27 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:27.857647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01febf7-f4e5-425e-9893-21a23ddd7f82 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.879120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aec1f196-ca98-403d-84de-6a22ba2aa5ec None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:27.886518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:27.922051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bce5398-21f3-4849-b514-864f9cf55751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:27.922875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce5398-21f3-4849-b514-864f9cf55751 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.923173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce5398-21f3-4849-b514-864f9cf55751 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.924049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bce5398-21f3-4849-b514-864f9cf55751 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.935349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bce5398-21f3-4849-b514-864f9cf55751 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:27.936789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bce5398-21f3-4849-b514-864f9cf55751 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-4bce5398-21f3-4849-b514-864f9cf55751 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:27.955191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:27.956146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:27.957028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:27.957028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:27.957028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:27.957028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:27.957028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 435/849] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:27.957982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f91931-6b1d-43d1-a1c2-ff14e152897c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:27.957982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f91931-6b1d-43d1-a1c2-ff14e152897c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:27.958244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f91931-6b1d-43d1-a1c2-ff14e152897c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:27.959628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 436/850] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:27.963666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f91931-6b1d-43d1-a1c2-ff14e152897c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:27.978670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d01febf7-f4e5-425e-9893-21a23ddd7f82 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.004600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:28.020428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a43172-d2c9-42a1-9dc6-d635bddc36a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.020428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a43172-d2c9-42a1-9dc6-d635bddc36a4 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.020428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a43172-d2c9-42a1-9dc6-d635bddc36a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.020428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a43172-d2c9-42a1-9dc6-d635bddc36a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.029438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a43172-d2c9-42a1-9dc6-d635bddc36a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e3a43172-d2c9-42a1-9dc6-d635bddc36a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:28.038084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:28.039078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:00:28.039907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:28.046955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 415/851] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:00:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:28.049418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984b3c3c-143b-4c88-8870-84d0c772dcc8 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.049722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984b3c3c-143b-4c88-8870-84d0c772dcc8 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.049998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-984b3c3c-143b-4c88-8870-84d0c772dcc8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.067553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-984b3c3c-143b-4c88-8870-84d0c772dcc8 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:28.068746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-984b3c3c-143b-4c88-8870-84d0c772dcc8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:28.079339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f91931-6b1d-43d1-a1c2-ff14e152897c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.123980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.124643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.125073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.125338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.126375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:28.127705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.129552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.142796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 416/852] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:28 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:28.142796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984b3c3c-143b-4c88-8870-84d0c772dcc8 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.160556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 437/853] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:28.161932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.184993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:28.252859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-984b3c3c-143b-4c88-8870-84d0c772dcc8 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.266620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:28.272443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6028b703-39e5-4cc7-a066-56ee4e6a6aef None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.287675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.288343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.288628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.288908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.289653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:28.289957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.291430 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.304574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 417/854] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:28.304574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.319224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.320012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.320291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.320565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] PATH_INFO: `/v3/users/swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.329302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.331882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:28.347220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:28.348181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:28.349115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 438/855] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/users/swift => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:28.349699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.407142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5da0aa-e57f-44c3-8d10-8cd1f444399d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.433223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:28.443020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.443695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.444007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.444486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.445656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.447041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.455587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-129fbb97-7c2c-4ecb-886e-ff8810d96dfa None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.489631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.490445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.490729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.491070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.501533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.504448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.511189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 418/856] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:28 2022] POST /identity/v3/services => generated 269 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:28.512194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:28.522671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:28.523828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:28.529894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:28.529894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:28.529894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:28.529894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:28.529894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 439/857] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:28.530210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.574547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:28.666668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b9bca24-345e-40bb-bbbc-2d3b0b8c8cc3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.678079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af53f4b1-7d49-4655-9660-53910a6d2121 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.718881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:28.723113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.723805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.724083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.724357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.725210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:28.725521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.727000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.732685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.735371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.735371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.735371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.735371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:28.735371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.736738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.748242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 419/858] 10.222.0.27 () {56 vars in 1135 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:28.753804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 440/859] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:28.755879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.759546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:28.860037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:28.873623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e70a57f-67c8-44fd-9931-bf9dd601554e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.896672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7577333-9ea6-4b9b-aa3a-6ef82a4e7248 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:28.912329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.913096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.913674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.913817 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.943295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:28.944299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:28.944651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:28.945029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:28.952713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:28.954336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:28.970065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:28.971646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:28.972568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:28.976872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 441/860] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:28.977735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.003714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.006190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.037333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.103665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40766e0a-11d5-4490-b6e6-73a55684b12d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.105897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.126293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 420/861] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/auth/tokens => generated 1261 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:00:29.127996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.141160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.142333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.142889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.143387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.144427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:29.144937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.146568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.169356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 442/862] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Jul 14 12:00:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:29.170608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.235723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ac97a6-920d-4394-9028-4bbc50d04b05 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.273288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.274095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.274221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be4b4af-b536-438d-9197-7454ea796b12 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.274403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.274943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.284905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.287471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.289104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.311236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.312040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.312436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.312809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:29.313042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:29.314151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:29.315178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:29.315178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:29.315178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:29.315178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:29.320986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 421/863] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:29.323704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.385690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.387131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.403278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.434598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9de2c47-f323-49b6-ad17-e52a4887fa47 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.473242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.474380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.474924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.475415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.476343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:29.476966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.478615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.495994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 443/864] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:29 2022] GET /identity/v3/auth/tokens => generated 1261 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:00:29.496175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 422/865] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:29.497646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.499016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.510596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.618962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.624024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4aceff31-8aab-4b60-b1fb-a084bf53c133 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.629707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6faa1546-ca2a-4ce7-8af8-04fc980dad56 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.666495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97887a9-1189-4cc7-974b-c88523738378 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.667465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97887a9-1189-4cc7-974b-c88523738378 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.667827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97887a9-1189-4cc7-974b-c88523738378 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.668452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97887a9-1189-4cc7-974b-c88523738378 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.677284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.678905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.679392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.680372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.681966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97887a9-1189-4cc7-974b-c88523738378 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.683710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a97887a9-1189-4cc7-974b-c88523738378 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.690052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.692629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-a97887a9-1189-4cc7-974b-c88523738378 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:29.705339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:29.706448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:29.707406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:29.707406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:29.707406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:29.707406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:29.707406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 423/866] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:29.708387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da34e51-db9f-4906-a5b3-75fe5947a797 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.708738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da34e51-db9f-4906-a5b3-75fe5947a797 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.709087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da34e51-db9f-4906-a5b3-75fe5947a797 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.710695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 424/867] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:29.713961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:29.715074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:29.716193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:29.723427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ed663d-f7b4-4e34-bc94-1c7f864c876b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.723427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ed663d-f7b4-4e34-bc94-1c7f864c876b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.723427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ed663d-f7b4-4e34-bc94-1c7f864c876b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.723720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 444/868] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:29.723720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.744439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f7ed663d-f7b4-4e34-bc94-1c7f864c876b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:29.746065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f7ed663d-f7b4-4e34-bc94-1c7f864c876b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:29.789094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.822845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e4b06e-677d-4d7b-ab2d-e30e5abd2308 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.828139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 425/869] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:29 2022] POST /identity/v3/auth/tokens => generated 1261 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:29.828818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ed663d-f7b4-4e34-bc94-1c7f864c876b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.877882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.878541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.878866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.879139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.879855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:29.880157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.881509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:29.894270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 445/870] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:29.895185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:29.928568 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:29.948673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ed663d-f7b4-4e34-bc94-1c7f864c876b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:29.984690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:29.985346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:29.985651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:29.986529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:29.987580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:29.987893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:29.989324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:30.001162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 426/871] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:30.002578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:30.009234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76bc65d5-7882-43ac-8b06-39a4a7583ee8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:30.039857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:30.047704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:30.048968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.049319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.049593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.050306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:30.051696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:30.105655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a5017ce-f898-4419-8ed9-720c2d4da8b5 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:30.139133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:30.139948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.140201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.140472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.145252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 446/872] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:29 2022] POST /identity/v3/services => generated 251 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:30.148327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:30.150042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:30.156389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:30.171051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:30.172023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:30.172981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 427/873] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:30.179594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:30.183632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:30.315847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b4cf0cc-3d7f-4291-8225-52da81166efc None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:30.320358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b515f5-9a39-455b-b5a3-1596dd9ee2e7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:30.364687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:30.365344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.365625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.365897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.366730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=7b1634a6e6ec4db9a8851b1329ffe6a0) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:30.367069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:30.368490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:30.372586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65b8f8a-a70f-40e2-a956-8ae67e71a3d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:30.373950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65b8f8a-a70f-40e2-a956-8ae67e71a3d0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.373950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65b8f8a-a70f-40e2-a956-8ae67e71a3d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.373950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65b8f8a-a70f-40e2-a956-8ae67e71a3d0 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.375461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65b8f8a-a70f-40e2-a956-8ae67e71a3d0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:30.375461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65b8f8a-a70f-40e2-a956-8ae67e71a3d0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:30.377098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 447/874] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:30 2022] GET /identity/v3/role_assignments?user.id=7b1634a6e6ec4db9a8851b1329ffe6a0&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:30.377217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e65b8f8a-a70f-40e2-a956-8ae67e71a3d0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:30.392865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 428/875] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:30.399008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:30.459165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:30.515983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5d070e-f38d-4b55-a31a-a74aaec27bb1 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:30.559548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010a7c04-afda-4dd3-9c53-fc231a70c4f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:30.560301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010a7c04-afda-4dd3-9c53-fc231a70c4f8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.560581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010a7c04-afda-4dd3-9c53-fc231a70c4f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.560855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010a7c04-afda-4dd3-9c53-fc231a70c4f8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.561687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010a7c04-afda-4dd3-9c53-fc231a70c4f8 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=1843ddb84e8b434dbed5ebd32419b8d6) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:30.561996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010a7c04-afda-4dd3-9c53-fc231a70c4f8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:30.563456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010a7c04-afda-4dd3-9c53-fc231a70c4f8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:30.570786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 448/876] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:30 2022] GET /identity/v3/role_assignments?user.id=1843ddb84e8b434dbed5ebd32419b8d6&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 608 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:30.644162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab279d-3884-4df6-9518-fa00afe32d0c None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.644503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab279d-3884-4df6-9518-fa00afe32d0c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.644727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdab279d-3884-4df6-9518-fa00afe32d0c None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.646470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 429/877] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:30.652487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee5ed27-3ec1-4e87-ab81-9cc9bc1f21a9 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.652813 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee5ed27-3ec1-4e87-ab81-9cc9bc1f21a9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.653039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee5ed27-3ec1-4e87-ab81-9cc9bc1f21a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.674352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0ee5ed27-3ec1-4e87-ab81-9cc9bc1f21a9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:30.675599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0ee5ed27-3ec1-4e87-ab81-9cc9bc1f21a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:30.753804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 449/878] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:30 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:30.777757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab279d-3884-4df6-9518-fa00afe32d0c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:30.850392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:30.895930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdab279d-3884-4df6-9518-fa00afe32d0c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:30.932087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:30.932911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:30.933197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:30.933474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:30.934139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:30.935578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:30.949084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:30.950074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:30.951206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 430/879] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Jul 14 12:00:30 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:30.959236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee5ed27-3ec1-4e87-ab81-9cc9bc1f21a9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:31.006180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:31.067619 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee5ed27-3ec1-4e87-ab81-9cc9bc1f21a9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:31.104141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:31.105238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:31.105767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:31.106278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:31.107227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:31.107740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:31.109381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:31.120240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 450/880] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:30 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:31.121823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:31.229716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb4ba8a-dd6e-4ec3-bbca-c19eb4aa2b63 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:31.257621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:31.266827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:31.268003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:31.268497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:31.269000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:31.269926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:31.270434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:31.272081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:31.277745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 431/881] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:31 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:31.284135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:31.393722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddefa2b4-5e46-4036-8fb5-171b61284159 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:31.401777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:31.429389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b59b453-d6b8-4968-8132-8bd2c106bf4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:31.430046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b59b453-d6b8-4968-8132-8bd2c106bf4d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:31.430325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b59b453-d6b8-4968-8132-8bd2c106bf4d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:31.430597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b59b453-d6b8-4968-8132-8bd2c106bf4d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:31.431415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b59b453-d6b8-4968-8132-8bd2c106bf4d None admin] RBAC: Adding query filter params (interface=public, service_id=a561d5ce6a8a4e35b3379d7184b4b228, region_id=RegionOne) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:31.431725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b59b453-d6b8-4968-8132-8bd2c106bf4d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:31.433100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b59b453-d6b8-4968-8132-8bd2c106bf4d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:31.438906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 451/882] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:00:31 2022] GET /identity/v3/endpoints?service_id=a561d5ce6a8a4e35b3379d7184b4b228&interface=public®ion_id=RegionOne => generated 194 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:31.444217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:31.551395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fdae16e-3b5f-45a5-b0f9-9495805b3e2d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:31.587452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb86a1a-4c2d-4b8d-a8c0-5491fee2a5f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:31.588115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb86a1a-4c2d-4b8d-a8c0-5491fee2a5f9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:31.589838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb86a1a-4c2d-4b8d-a8c0-5491fee2a5f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:31.589838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb86a1a-4c2d-4b8d-a8c0-5491fee2a5f9 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:31.589838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb86a1a-4c2d-4b8d-a8c0-5491fee2a5f9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:31.590731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeb86a1a-4c2d-4b8d-a8c0-5491fee2a5f9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:31.597823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 432/883] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:31 2022] GET /identity/v3/services => generated 1268 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:00:32.313529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeaca62-0532-41e6-b14d-3671edaa1d2f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.315216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeaca62-0532-41e6-b14d-3671edaa1d2f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.315714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aeaca62-0532-41e6-b14d-3671edaa1d2f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.317592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 452/884] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:32.323012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3dc47b-f4f2-400d-86df-55fbbf0672a0 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.323853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3dc47b-f4f2-400d-86df-55fbbf0672a0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.324336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3dc47b-f4f2-400d-86df-55fbbf0672a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.342055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ff3dc47b-f4f2-400d-86df-55fbbf0672a0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:32.343610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ff3dc47b-f4f2-400d-86df-55fbbf0672a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:32.395451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:32.414478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 433/885] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:32 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:32.438632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeaca62-0532-41e6-b14d-3671edaa1d2f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:32.548903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aeaca62-0532-41e6-b14d-3671edaa1d2f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:32.579216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd99c857-b8b5-453f-85e4-c23c7fda5382 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.580410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd99c857-b8b5-453f-85e4-c23c7fda5382 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.580921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd99c857-b8b5-453f-85e4-c23c7fda5382 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.583942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 434/886] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:32 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:32.586752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:32.587767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae65033-82b2-4c18-b8a9-47a091943cfa None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.588022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.588144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae65033-82b2-4c18-b8a9-47a091943cfa None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.588394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ae65033-82b2-4c18-b8a9-47a091943cfa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.588814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.589281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.590203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:32.591985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:32.604571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:32.605665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:32.606492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:32.607250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7ae65033-82b2-4c18-b8a9-47a091943cfa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:32.607250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7ae65033-82b2-4c18-b8a9-47a091943cfa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:32.609480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 453/887] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:32 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:32.615896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:32.653111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:32.693935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 435/888] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:32 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:32.727920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae65033-82b2-4c18-b8a9-47a091943cfa None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:32.732343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ff37f6-bdcb-4982-b4ff-8936d7a6fc8c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:32.776520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:32.777673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.778629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.779134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.780452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:32.781185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:32.784353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:32.791582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 454/889] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:32 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:32.796580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:32.846221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ae65033-82b2-4c18-b8a9-47a091943cfa None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:32.862675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:32.885760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17f0a41-8b51-475b-8dd8-63bce5530fb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:32.886588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f0a41-8b51-475b-8dd8-63bce5530fb5 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.886905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f0a41-8b51-475b-8dd8-63bce5530fb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.887179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17f0a41-8b51-475b-8dd8-63bce5530fb5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.887938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f0a41-8b51-475b-8dd8-63bce5530fb5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:32.889328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17f0a41-8b51-475b-8dd8-63bce5530fb5 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:32.900777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-c17f0a41-8b51-475b-8dd8-63bce5530fb5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:32.902351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:32.903495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:32.904330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:32.905087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b0d34-c990-4cd2-8d94-633ee853ff11 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:32.916436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 436/890] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:32 2022] GET /identity/v3/services/object-store => generated 93 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:32.916436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fd2970-c52e-40ee-9147-c9bea529228d None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.916436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fd2970-c52e-40ee-9147-c9bea529228d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.916436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fd2970-c52e-40ee-9147-c9bea529228d None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.916436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 437/891] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:32.916436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fd2970-c52e-40ee-9147-c9bea529228d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:32.945724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e553cb2e-8c7b-4f6d-9ac6-72c089b4bcd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:32.946392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553cb2e-8c7b-4f6d-9ac6-72c089b4bcd9 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.946675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553cb2e-8c7b-4f6d-9ac6-72c089b4bcd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.947129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e553cb2e-8c7b-4f6d-9ac6-72c089b4bcd9 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.947851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e553cb2e-8c7b-4f6d-9ac6-72c089b4bcd9 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:32.948159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e553cb2e-8c7b-4f6d-9ac6-72c089b4bcd9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:32.949522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e553cb2e-8c7b-4f6d-9ac6-72c089b4bcd9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:32.958794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 455/892] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:32 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:32.958794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c42835-53fb-47eb-a847-c363ef939d24 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:32.958794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c42835-53fb-47eb-a847-c363ef939d24 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:32.958794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c42835-53fb-47eb-a847-c363ef939d24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:32.972751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b2c42835-53fb-47eb-a847-c363ef939d24 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:32.973936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b2c42835-53fb-47eb-a847-c363ef939d24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:33.022040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fd2970-c52e-40ee-9147-c9bea529228d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.050411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 456/893] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:32 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:33.051532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c42835-53fb-47eb-a847-c363ef939d24 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.070750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.072026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.072547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.073421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.074363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:33.074904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:33.077276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:33.085221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 438/894] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:33.086916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.133364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:33.151168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:33.161037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c42835-53fb-47eb-a847-c363ef939d24 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.195446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3484b5c-9aee-4751-adee-e916fd54f1ee None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.195929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.196590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.196878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.197149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.197942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] RBAC: Adding query filter params (interface=public, service_id=d00cb056a2ea43f980a4340ed7e31f76, region_id=RegionOne) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:33.198254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:33.199654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:33.205383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 457/895] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/endpoints?service_id=d00cb056a2ea43f980a4340ed7e31f76&interface=public®ion_id=RegionOne => generated 194 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:33.206258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.234584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.235451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.236505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.236739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.247778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:33.256455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:33.257449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:33.258297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 439/896] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:33.259096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.323587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c758cf-02d6-4692-a945-fbc5791106f7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.358548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.359272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.359550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.359821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.360520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:33.360945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:33.362393 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:33.367151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 458/897] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:33.367814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.382747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b71641-b5cc-4ad4-8b96-90436674ce9e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.387453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:33.409372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:33.425495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.426141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.426421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.426703 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.427438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:33.429075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:33.443386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 440/898] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/services => generated 1268 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:00:33.511393 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a6f866-1a03-49d3-a837-bef05c9ac38f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.553102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-238ebbfa-8e06-41dd-9e9e-044a1febc149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.553789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238ebbfa-8e06-41dd-9e9e-044a1febc149 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.554076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238ebbfa-8e06-41dd-9e9e-044a1febc149 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.554405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-238ebbfa-8e06-41dd-9e9e-044a1febc149 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.555158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238ebbfa-8e06-41dd-9e9e-044a1febc149 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:33.555468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238ebbfa-8e06-41dd-9e9e-044a1febc149 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:33.556886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-238ebbfa-8e06-41dd-9e9e-044a1febc149 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:33.567186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 459/899] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:33.572947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.685138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d21b640-ab45-4ea6-9759-58cabddc874b None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.707407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1513160f-e1dd-4f0b-b3d6-f7fff86a309e None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.707694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1513160f-e1dd-4f0b-b3d6-f7fff86a309e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.707977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1513160f-e1dd-4f0b-b3d6-f7fff86a309e None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.710529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 460/900] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:33.714623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba72c84-54a7-42be-9b97-7783fd10374b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.714946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba72c84-54a7-42be-9b97-7783fd10374b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.715229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba72c84-54a7-42be-9b97-7783fd10374b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.729964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.730797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.731097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.731376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.739338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2ba72c84-54a7-42be-9b97-7783fd10374b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:33.740823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2ba72c84-54a7-42be-9b97-7783fd10374b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:33.745270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:33.746732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:33.770345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:33.771336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:33.772622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:33.772622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:33.772622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:33.772622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:33.772622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 441/901] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:33.777613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.799875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:33.828850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 461/902] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:33 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:33.855680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba72c84-54a7-42be-9b97-7783fd10374b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.876190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:33.901507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d02b3b-b038-43b6-bb1b-f789071b8ce7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:33.938934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:33.939985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:33.940546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:33.941026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:33.941973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:33.942486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:33.944260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:33.957528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 442/903] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:33.963298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:33.981558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba72c84-54a7-42be-9b97-7783fd10374b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.019552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.020364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.020644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.020925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.021580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.023205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:34.037344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:34.038321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 14 12:00:34.039194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:34.043514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 462/904] 10.222.0.27 () {56 vars in 1132 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:34.044905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:34.053696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:34.076475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14347b65-f5a2-4642-948a-7abefb1ffc6a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.114000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.115056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.115475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.115856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.123831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.125529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.134920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:34.142862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:34.148927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:34.150847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 443/905] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:00:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:34.151453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:34.164860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c1947bc-e578-4eae-ac29-f5f087fb187d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.202863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.203879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.204953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.205440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.206381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:34.206920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.208706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.218662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 463/906] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:34 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:34.220686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:34.263080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5125ef87-76c7-48fd-8672-bd3214a1c12f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.303443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.304194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.304478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.304754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.305809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:34.306094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.307668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.316226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:34.326878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 444/907] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Jul 14 12:00:34 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:34.330275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1ebf1c-9409-44d4-a993-f9977f7716fe None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.343605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:34.367724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.368683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.369170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.369781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.370818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:34.371330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.372994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.383521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 464/908] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:34 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:34.385034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:34.389369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:34.450289 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69a522d-11ef-4930-96ea-1ca863cd1804 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.488362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.488977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.489221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.489616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.496405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb466f1-880a-4a78-b9d6-4b25ff529493 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.533578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.534816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.535341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.535914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.537004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.538803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.562220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:34.569491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.571068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.614407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 465/909] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:34 2022] POST /identity/v3/endpoints => generated 347 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:00:34.692435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 445/910] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:34 2022] GET /identity/v3/auth/tokens => generated 1362 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:00:34.698193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:34.775948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:34.824852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff3d9f1-3db8-4ede-b3e0-f64500eaddee None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:34.860899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:34.861793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:34.862075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:34.862348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:34.873541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:34.875018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:34.893995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:34.895117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:34.895997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:34.895997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:34.895997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:34.895997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:34.895997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 466/911] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:34.902486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:35.007151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:35.009588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5483124-9ecb-4698-a999-289dce64260e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:35.046611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:35.047235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.047518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.047799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.049299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:35.049611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:35.051004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:35.063363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 446/912] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:35.071863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:35.185460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184234a9-0b44-4596-b58d-9d991a029173 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:35.196041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:35.223239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:35.224425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.224917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.225401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.232967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:35.234738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:35.250477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:35.251477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:35.252372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:35.258879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 467/913] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:35.260827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:35.365302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35c1db3-cdea-49b0-ba83-40ce7ee07bb8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:35.399489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7467176b-6b42-437c-89ab-9e59b8e72177 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:35.400407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7467176b-6b42-437c-89ab-9e59b8e72177 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.400689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7467176b-6b42-437c-89ab-9e59b8e72177 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.400963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7467176b-6b42-437c-89ab-9e59b8e72177 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.401749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7467176b-6b42-437c-89ab-9e59b8e72177 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:35.402058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7467176b-6b42-437c-89ab-9e59b8e72177 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:35.403507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7467176b-6b42-437c-89ab-9e59b8e72177 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:35.416717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 447/914] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:35 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:35.422490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:35.485366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee98d92-80d5-4cb6-96a0-a944b3e463f7 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.486597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee98d92-80d5-4cb6-96a0-a944b3e463f7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.486597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee98d92-80d5-4cb6-96a0-a944b3e463f7 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.488316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 448/915] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:35.493570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443ecdb-3397-4bce-b3b9-caeb613ada94 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.494092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443ecdb-3397-4bce-b3b9-caeb613ada94 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.494576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7443ecdb-3397-4bce-b3b9-caeb613ada94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.512194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7443ecdb-3397-4bce-b3b9-caeb613ada94 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:35.513845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7443ecdb-3397-4bce-b3b9-caeb613ada94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:35.527521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:35.544086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc77ec61-3974-4b06-afbe-7de249541c50 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:35.590560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:35.591259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.591543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.591820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.592663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] RBAC: Adding query filter params (role.id=bc3dc6d5822c488dafecfc5fbf99ef8b, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=1843ddb84e8b434dbed5ebd32419b8d6) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:35.593003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:35.594440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:35.598203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 449/916] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:35 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (3 switches on core 0) Jul 14 12:00:35.599735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a334e7-3d38-4e04-82c8-f063a6e040c7 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.600025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a334e7-3d38-4e04-82c8-f063a6e040c7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.600302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a334e7-3d38-4e04-82c8-f063a6e040c7 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.601696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 450/917] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:35.606022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 468/918] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:35 2022] GET /identity/v3/role_assignments?user.id=1843ddb84e8b434dbed5ebd32419b8d6&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=bc3dc6d5822c488dafecfc5fbf99ef8b => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:35.606524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a595b0-fccd-4d38-ba71-53b3ccba63e7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.606849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a595b0-fccd-4d38-ba71-53b3ccba63e7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.607136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a595b0-fccd-4d38-ba71-53b3ccba63e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.622881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:35.628402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-33a595b0-fccd-4d38-ba71-53b3ccba63e7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:35.629705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-33a595b0-fccd-4d38-ba71-53b3ccba63e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:35.666179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:35.716247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 451/919] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:35 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:35.737804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a595b0-fccd-4d38-ba71-53b3ccba63e7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:35.753383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1f0284-659e-4a97-8ac6-a080e88bb033 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:35.798060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:35.799608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.799940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.800222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.800968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:35.802465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:35.826931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:35.849742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a595b0-fccd-4d38-ba71-53b3ccba63e7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:35.879419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 469/920] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:35 2022] POST /identity/v3/services => generated 241 bytes in 251 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:00:35.896583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:35.897411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:35.897686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:35.897965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:35.898635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:35.900464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:35.913712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:35.914931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:35.915884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 452/921] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:35 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:35.924885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:36.028510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:36.038316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe29845b-3aca-4740-a1c6-9321ba99c679 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:36.078840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:36.079687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:36.079971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:36.080289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:36.081022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:36.081326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:36.082698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:36.088451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 470/922] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:35 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:36.093767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:36.201009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08da6eea-4adf-43be-86eb-336a4685fab8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:36.240730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1d6ca6-8d62-449e-a32b-5e2215edbaa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:36.241398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1d6ca6-8d62-449e-a32b-5e2215edbaa8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:36.241679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1d6ca6-8d62-449e-a32b-5e2215edbaa8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:36.241952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1d6ca6-8d62-449e-a32b-5e2215edbaa8 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:36.242662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1d6ca6-8d62-449e-a32b-5e2215edbaa8 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:36.242997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1d6ca6-8d62-449e-a32b-5e2215edbaa8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:36.244356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a1d6ca6-8d62-449e-a32b-5e2215edbaa8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:36.246519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:36.249768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 453/923] 10.222.0.27 () {56 vars in 1114 bytes} [Thu Jul 14 12:00:36 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:36.255727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:36.364294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9baaf48-9ca2-4ba3-9250-89bfe48901b8 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:36.401561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6ec581-cf62-4133-9235-a6a75b2ba2a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:36.402225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6ec581-cf62-4133-9235-a6a75b2ba2a9 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:36.402506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6ec581-cf62-4133-9235-a6a75b2ba2a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:36.402801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6ec581-cf62-4133-9235-a6a75b2ba2a9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:36.403562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6ec581-cf62-4133-9235-a6a75b2ba2a9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:36.404943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec6ec581-cf62-4133-9235-a6a75b2ba2a9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:36.495771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 471/924] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:36 2022] POST /identity/v3/endpoints => generated 346 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:36.916803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14580e66-87d9-446a-969b-d2f694e625a5 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:36.917501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14580e66-87d9-446a-969b-d2f694e625a5 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:36.917501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14580e66-87d9-446a-969b-d2f694e625a5 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:36.918916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 454/925] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:36.924788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cf9bfb-14e1-4004-8545-651b45184180 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:36.925082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cf9bfb-14e1-4004-8545-651b45184180 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:36.925362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55cf9bfb-14e1-4004-8545-651b45184180 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:36.943250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-55cf9bfb-14e1-4004-8545-651b45184180 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:36.944447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-55cf9bfb-14e1-4004-8545-651b45184180 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:36.952758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:37.020279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 472/926] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:36 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:37.043261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14580e66-87d9-446a-969b-d2f694e625a5 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:37.201757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14580e66-87d9-446a-969b-d2f694e625a5 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:37.263140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:37.264465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:37.264957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:37.265451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:37.266745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:37.269688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:37.292580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:37.296051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:37.297827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 455/927] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:37 2022] GET /identity/v3/services/compute => generated 88 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:37.303851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cf9bfb-14e1-4004-8545-651b45184180 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:37.412530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55cf9bfb-14e1-4004-8545-651b45184180 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:37.448703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbd1af-80b6-4d2e-91e6-307ef72b7276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:37.449758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbd1af-80b6-4d2e-91e6-307ef72b7276 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:37.450302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbd1af-80b6-4d2e-91e6-307ef72b7276 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:37.450809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbd1af-80b6-4d2e-91e6-307ef72b7276 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:37.451727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbd1af-80b6-4d2e-91e6-307ef72b7276 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:37.452300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbd1af-80b6-4d2e-91e6-307ef72b7276 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:37.453897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbd1af-80b6-4d2e-91e6-307ef72b7276 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:37.460154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 473/928] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:37 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:37.467852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:37.531101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:37.570677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a75a88-9915-41f7-b543-20430b3c3de0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:37.610737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e4762f-ded4-411b-86de-0ad33c78212f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:37.611884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e4762f-ded4-411b-86de-0ad33c78212f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:37.612384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e4762f-ded4-411b-86de-0ad33c78212f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:37.612877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e4762f-ded4-411b-86de-0ad33c78212f None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:37.614043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e4762f-ded4-411b-86de-0ad33c78212f None admin] RBAC: Adding query filter params (type=compute) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:37.614576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e4762f-ded4-411b-86de-0ad33c78212f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:37.616193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e4762f-ded4-411b-86de-0ad33c78212f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:37.621874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 456/929] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:37 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:37.724975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aa4005-68ad-4cc4-a702-4a0cd853888c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:37.725946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aa4005-68ad-4cc4-a702-4a0cd853888c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:37.726444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0aa4005-68ad-4cc4-a702-4a0cd853888c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:37.728330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 474/930] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:37.733911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0446f2-423b-41b0-803a-81254d5397c5 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:37.734199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0446f2-423b-41b0-803a-81254d5397c5 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:37.734478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0446f2-423b-41b0-803a-81254d5397c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:37.753777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3b0446f2-423b-41b0-803a-81254d5397c5 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:37.755036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3b0446f2-423b-41b0-803a-81254d5397c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:37.829153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 457/931] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:37 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:37.851293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aa4005-68ad-4cc4-a702-4a0cd853888c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:37.854471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:37.964613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0aa4005-68ad-4cc4-a702-4a0cd853888c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.001224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff1c706-807a-4c90-9015-a8cc317ff335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.002013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1c706-807a-4c90-9015-a8cc317ff335 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.002295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1c706-807a-4c90-9015-a8cc317ff335 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.002574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff1c706-807a-4c90-9015-a8cc317ff335 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.013783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff1c706-807a-4c90-9015-a8cc317ff335 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-fff1c706-807a-4c90-9015-a8cc317ff335 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:38.022733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:38.023700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:38.024554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 475/932] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Jul 14 12:00:37 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:38.030109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0446f2-423b-41b0-803a-81254d5397c5 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.089776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:38.133496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0446f2-423b-41b0-803a-81254d5397c5 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.149271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ab763b-7cf3-4bde-ac94-e757b4f023dd None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.149590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ab763b-7cf3-4bde-ac94-e757b4f023dd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.150094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ab763b-7cf3-4bde-ac94-e757b4f023dd None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.151786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 476/933] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:38.156777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10a6784-b9b1-491b-af66-ded25385197e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.157066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10a6784-b9b1-491b-af66-ded25385197e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.157420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10a6784-b9b1-491b-af66-ded25385197e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.175428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.176474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.176993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.177471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.177854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a10a6784-b9b1-491b-af66-ded25385197e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:38.178668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:38.179101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a10a6784-b9b1-491b-af66-ded25385197e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:38.179446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.181141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.190801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 458/934] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:38.202795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.254035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:38.270846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 477/935] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:38 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:38.295938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10a6784-b9b1-491b-af66-ded25385197e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.322703 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ada74a43-7682-409a-b826-e303b3fe0056 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.344729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:38.361598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.362808 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.363289 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.363749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.376241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.378388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:38.397984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:38.398970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:38.400190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:38.400190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:38.400190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:38.400190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:38.402505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 459/936] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:38.406961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10a6784-b9b1-491b-af66-ded25385197e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.409059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.443891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.444792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.445141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.445483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.446208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.448276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:38.461172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:38.462421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:38.463484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:38.470460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 478/937] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:38.471829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.522051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b6e6993-f478-46e8-82da-bed2303fc7d5 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.533221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:38.559113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.560062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.560514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.560982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.561871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:38.562351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.564470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.581342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6425f24-ac6f-450a-ac2a-ce77991db687 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.587218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 460/938] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:38.588612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.618445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.619147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.619498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.619697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.620497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:38.620836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.622313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.622888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:38.633736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 479/939] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:38.642911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.705359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6465b23-28c0-4e9c-8590-991e1dda8e4a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.743667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d814877-dff4-4e77-99e0-39e67bc1188f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.744639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d814877-dff4-4e77-99e0-39e67bc1188f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.745246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d814877-dff4-4e77-99e0-39e67bc1188f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.745530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d814877-dff4-4e77-99e0-39e67bc1188f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.753454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60369083-8344-41cf-b1a3-c5f693e6d80c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.753933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d814877-dff4-4e77-99e0-39e67bc1188f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.755395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d814877-dff4-4e77-99e0-39e67bc1188f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-0d814877-dff4-4e77-99e0-39e67bc1188f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:38.771881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:38.773094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:38.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 461/940] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:38.779236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:38.779458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0337ea1-8b18-4a1f-8335-096af404bddf None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.779707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0337ea1-8b18-4a1f-8335-096af404bddf None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.779982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0337ea1-8b18-4a1f-8335-096af404bddf None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.781277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 462/941] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:38 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:38.781982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0337ea1-8b18-4a1f-8335-096af404bddf None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.799801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ad5ea6-c389-4e17-847f-48f9db547bf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.800909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ad5ea6-c389-4e17-847f-48f9db547bf1 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.801495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ad5ea6-c389-4e17-847f-48f9db547bf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.801944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ad5ea6-c389-4e17-847f-48f9db547bf1 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.802846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ad5ea6-c389-4e17-847f-48f9db547bf1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:38.803326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ad5ea6-c389-4e17-847f-48f9db547bf1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.804961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ad5ea6-c389-4e17-847f-48f9db547bf1 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.811536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 480/942] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:38.813062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836e1d4-3259-49d0-b5df-b26052975c12 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.813506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836e1d4-3259-49d0-b5df-b26052975c12 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.813966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3836e1d4-3259-49d0-b5df-b26052975c12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.833299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3836e1d4-3259-49d0-b5df-b26052975c12 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:38.834890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3836e1d4-3259-49d0-b5df-b26052975c12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:38.881913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:38.891354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0337ea1-8b18-4a1f-8335-096af404bddf None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:38.906868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 481/943] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:38 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:38.909270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836e1d4-3259-49d0-b5df-b26052975c12 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:38.927300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:38.928115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:38.928520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:38.928961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:38.929874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:38.930259 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:38.931790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:38.949424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 463/944] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:38.950299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.028875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3836e1d4-3259-49d0-b5df-b26052975c12 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.043809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:39.053785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7dd4ebb-41f4-4225-ba0a-a4b60638400c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.065982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.066722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.067036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.067315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.068104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] RBAC: Adding query filter params (interface=public, service_id=5c738d39141847b8b8b0b025cbb8ed76, region_id=RegionOne) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:39.068420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.070022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.079806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 482/945] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:00:38 2022] GET /identity/v3/endpoints?service_id=5c738d39141847b8b8b0b025cbb8ed76&interface=public®ion_id=RegionOne => generated 194 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:39.079806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.094251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.095691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.095949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.096204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.096903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.098320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.099698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING keystone.server.flask.common [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 14 12:00:39.132829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.sql.core [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 14 12:00:39.132829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 14 12:00:39.132829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [parameters: {'id': 'e816cb50691f4a59ab8d1aaddb942aec', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 14 12:00:39.132829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80488) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e816cb50691f4a59ab8d1aaddb942aec.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e816cb50691f4a59ab8d1aaddb942aec. Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application result = self._query(query) Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application conn.query(q) Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 14 12:00:39.156621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application result.read() Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 14 12:00:39.157472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application next(self.gen) Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application yield resource Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application next(self.gen) Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self.session.rollback() Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self.force_reraise() Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise self.value Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application session.commit() Jul 14 12:00:39.158454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._prepare_impl() Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self.session.flush() Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._flush(objects) Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application compat.raise_( Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application flush_context.execute() Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 14 12:00:39.160307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rec.execute(self) Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 14 12:00:39.161466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application result = self._query(query) Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application conn.query(q) Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application result.read() Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 14 12:00:39.162262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [parameters: {'id': 'e816cb50691f4a59ab8d1aaddb942aec', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:39.164173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:39.165825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id e816cb50691f4a59ab8d1aaddb942aec. Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 464/946] 10.222.0.27 () {60 vars in 1140 bytes} [Thu Jul 14 12:00:38 2022] POST /identity/v3/projects => generated 262 bytes in 209 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:39.167113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.178587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:39.187947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb0e395-e169-43a9-a99c-4e714eecbab5 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.224002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.224929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.225312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.225773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.236266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.237766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:39.256544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:39.257658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:39.258544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:39.258544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:39.258544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:39.258544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:39.258544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 483/947] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:39.258544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.284914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0052b23-bb8e-4ea4-aaf0-2aa05ad08706 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.291043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:39.321412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.322086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.322371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.322649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.323368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.324936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.334996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 465/948] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:39 2022] GET /identity/v3/services => generated 1497 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:00:39.343827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.382950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09785a2f-160a-4908-a1f1-2bf3651a42fd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.425964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:39.435936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.436793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.437077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.437359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.446517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.447984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:39.463354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:39.464331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:39.465227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:39.470053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 484/949] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:39.470933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77ceb9b-43ed-4e3e-af46-02b636c9fec1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.471115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.515597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.516271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.516556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.516948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.517816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:39.518131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.519572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.532950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 466/950] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:39.538578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.583123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:39.601834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8589a5d-51c8-428f-8645-794b2181e915 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.651274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.652549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.653016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.653711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.655217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:39.655841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.657416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecb26260-cb44-4a60-a050-11807c6baa81 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.657824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.682043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 485/951] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:39 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:39.697847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:39.705650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e916409c-3fab-44b3-af93-69f861889f1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.708665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916409c-3fab-44b3-af93-69f861889f1a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.708665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916409c-3fab-44b3-af93-69f861889f1a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.708665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e916409c-3fab-44b3-af93-69f861889f1a None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.715080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e916409c-3fab-44b3-af93-69f861889f1a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.716552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e916409c-3fab-44b3-af93-69f861889f1a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e916409c-3fab-44b3-af93-69f861889f1a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:39.732641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:39.733698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:39.735091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 467/952] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:39.740627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.784258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9286250-622c-4e1c-ae1f-ee2f12eca96e None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.784538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9286250-622c-4e1c-ae1f-ee2f12eca96e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.784816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9286250-622c-4e1c-ae1f-ee2f12eca96e None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.786244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 468/953] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:39.792058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f9250a-424e-4303-af80-c45d413b74ee None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.792298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f9250a-424e-4303-af80-c45d413b74ee None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.792708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f9250a-424e-4303-af80-c45d413b74ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.811761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-62f9250a-424e-4303-af80-c45d413b74ee None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:39.813410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-62f9250a-424e-4303-af80-c45d413b74ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:39.867529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f3e62f-a5fb-4f15-ba92-19f08b6aa507 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:39.909122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 469/954] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:39 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:39.915742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:39.916611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:39.917089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:39.917549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:39.918508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:39.919050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:39.920640 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:39.932108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f9250a-424e-4303-af80-c45d413b74ee None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.933399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 486/955] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:39.939441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:39.953974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:40.057007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-874ca4e3-e8bb-4afb-b6a4-245691841ccf None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:40.064153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f9250a-424e-4303-af80-c45d413b74ee None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:40.092561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:40.097571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f730a1-6c7a-4d5c-aeb2-262f25ab8adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:40.099286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f730a1-6c7a-4d5c-aeb2-262f25ab8adb None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:40.099762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f730a1-6c7a-4d5c-aeb2-262f25ab8adb None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:40.100371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f730a1-6c7a-4d5c-aeb2-262f25ab8adb None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/1843ddb84e8b434dbed5ebd32419b8d6/roles/bc3dc6d5822c488dafecfc5fbf99ef8b` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:40.103521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f730a1-6c7a-4d5c-aeb2-262f25ab8adb None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=1843ddb84e8b434dbed5ebd32419b8d6, role_id=bc3dc6d5822c488dafecfc5fbf99ef8b)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:40.105734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f730a1-6c7a-4d5c-aeb2-262f25ab8adb None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:40.109541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a780a6-9323-4a3f-9fcf-9c93298b7c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:40.110365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a780a6-9323-4a3f-9fcf-9c93298b7c3b None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:40.112840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a780a6-9323-4a3f-9fcf-9c93298b7c3b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:40.112840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a780a6-9323-4a3f-9fcf-9c93298b7c3b None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:40.112840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a780a6-9323-4a3f-9fcf-9c93298b7c3b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:40.113599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a780a6-9323-4a3f-9fcf-9c93298b7c3b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:40.251961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 487/956] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:39 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/1843ddb84e8b434dbed5ebd32419b8d6/roles/bc3dc6d5822c488dafecfc5fbf99ef8b => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:00:40.256321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 470/957] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:39 2022] POST /identity/v3/services => generated 242 bytes in 325 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:00:41.451708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c02969e-0796-4c4c-9931-40d6ce041443 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:41.452240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c02969e-0796-4c4c-9931-40d6ce041443 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:41.452240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c02969e-0796-4c4c-9931-40d6ce041443 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:41.453919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 488/958] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:41.459245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee044c-98af-4287-bbd3-5444facfb471 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:41.459698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee044c-98af-4287-bbd3-5444facfb471 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:41.459876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbee044c-98af-4287-bbd3-5444facfb471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:41.478995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-dbee044c-98af-4287-bbd3-5444facfb471 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:41.480205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dbee044c-98af-4287-bbd3-5444facfb471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:41.566791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 471/959] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:41 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:41.584701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c02969e-0796-4c4c-9931-40d6ce041443 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:41.659821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:41.692346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c02969e-0796-4c4c-9931-40d6ce041443 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:41.727769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:41.728586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:41.728865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:41.729144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:41.729887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:41.731287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:41.744111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:41.745261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:41.746209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 489/960] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:41.755868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee044c-98af-4287-bbd3-5444facfb471 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:41.859569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbee044c-98af-4287-bbd3-5444facfb471 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:41.867778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:41.897187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f6a9f3-2745-4a48-8585-4460a387481e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:41.897862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f6a9f3-2745-4a48-8585-4460a387481e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:41.898144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f6a9f3-2745-4a48-8585-4460a387481e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:41.898414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f6a9f3-2745-4a48-8585-4460a387481e None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:41.899150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f6a9f3-2745-4a48-8585-4460a387481e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:41.899453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f6a9f3-2745-4a48-8585-4460a387481e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:41.901065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f6a9f3-2745-4a48-8585-4460a387481e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:41.906636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 472/961] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:41.914402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:41.966201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2d8460-29fd-46a8-b91f-e5a68d5953dd None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:41.966620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2d8460-29fd-46a8-b91f-e5a68d5953dd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:41.967076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2d8460-29fd-46a8-b91f-e5a68d5953dd None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:41.969183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 473/962] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:41.975878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd73aa-6271-404f-84f9-de58955f0a00 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:41.976293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd73aa-6271-404f-84f9-de58955f0a00 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:41.976765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bd73aa-6271-404f-84f9-de58955f0a00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:41.995275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e5bd73aa-6271-404f-84f9-de58955f0a00 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:41.996680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e5bd73aa-6271-404f-84f9-de58955f0a00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:42.036903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334f1284-6624-498d-87a3-fc3a857427e7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.072736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 474/963] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:41 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:42.077620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.078282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.078560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.078865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.080400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:42.080709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:42.082097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:42.087781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:42.091646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 490/964] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:42.097552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bd73aa-6271-404f-84f9-de58955f0a00 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.100353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.207895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:42.211139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bd73aa-6271-404f-84f9-de58955f0a00 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.214285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76bbae9-0056-4f6a-955c-df2cd381da42 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.248801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.249381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.249662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.249939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.250715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:42.252814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:42.253805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.254787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.255330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.255810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.256782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:42.258493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:42.260223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING keystone.server.flask.common [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 14 12:00:42.309848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.sql.core [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 14 12:00:42.309848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 14 12:00:42.309848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [parameters: {'user_id': '87261c3c78b84a1ea1aafa1c54fe3d3d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 14 12:00:42.309848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80487) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 14 12:00:42.318820 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 475/965] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:42 2022] POST /identity/v3/endpoints => generated 349 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-b8739cef-8d92-4c1c-83ee-2a37aa0ee836 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result = self._query(query) Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application conn.query(q) Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result.read() Jul 14 12:00:42.323453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 14 12:00:42.324392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application next(self.gen) Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application yield resource Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application next(self.gen) Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.session.rollback() Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.force_reraise() Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise self.value Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application session.commit() Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 14 12:00:42.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._prepare_impl() Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.session.flush() Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._flush(objects) Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application compat.raise_( Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application flush_context.execute() Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rec.execute(self) Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 14 12:00:42.326358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 14 12:00:42.328024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result = self._query(query) Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application conn.query(q) Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application result.read() Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 14 12:00:42.328896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [parameters: {'user_id': '87261c3c78b84a1ea1aafa1c54fe3d3d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:42.330412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 14 12:00:42.331299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 14 12:00:42.333969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 14 12:00:42.333969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 14 12:00:42.333969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 14 12:00:42.333969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 14 12:00:42.333969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 14 12:00:42.333969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:42.336835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 491/966] 10.222.0.27 () {60 vars in 1131 bytes} [Thu Jul 14 12:00:42 2022] POST /identity/v3/users => generated 160 bytes in 237 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:42.343061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.344637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6381fd4d-a3d9-432c-b392-3f43a6bdf1b0 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.345047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6381fd4d-a3d9-432c-b392-3f43a6bdf1b0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.345334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6381fd4d-a3d9-432c-b392-3f43a6bdf1b0 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.351199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 492/967] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:42 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:42.356678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b808b9-d6b1-432e-8e64-0d0a043d3583 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.356964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b808b9-d6b1-432e-8e64-0d0a043d3583 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.357246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b808b9-d6b1-432e-8e64-0d0a043d3583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.375666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-82b808b9-d6b1-432e-8e64-0d0a043d3583 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:42.377830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-82b808b9-d6b1-432e-8e64-0d0a043d3583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:42.462055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:42.482948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 493/968] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:42 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:42.482948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedc6d1a-fe23-4e5d-aa76-db9d5a1ddb50 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.482948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedc6d1a-fe23-4e5d-aa76-db9d5a1ddb50 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.482948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedc6d1a-fe23-4e5d-aa76-db9d5a1ddb50 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.488034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 494/969] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:42.488123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7a46ea-6ac5-4c9a-b391-4d6dfb2cfd75 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.490318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33686a3-c481-4fb8-8643-c14b1f33680b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.494822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33686a3-c481-4fb8-8643-c14b1f33680b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.495158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33686a3-c481-4fb8-8643-c14b1f33680b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.516183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c33686a3-c481-4fb8-8643-c14b1f33680b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:42.517422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c33686a3-c481-4fb8-8643-c14b1f33680b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:42.528690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.529492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.529765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.530036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.538224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:42.539852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:42.556346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:42.557327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 476/970] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:42 2022] GET /identity/v3/users/cinder => generated 84 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:42.558896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.577647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:42.595991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 495/971] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:42 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:42.596833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33686a3-c481-4fb8-8643-c14b1f33680b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.670660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69173d2-9cac-46c1-bae9-906e8645908c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.703362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33686a3-c481-4fb8-8643-c14b1f33680b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.711452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.712276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.712560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.712839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.726187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:42.730213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:42.738530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:42.741752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 477/972] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Jul 14 12:00:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:42.742609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.753306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.753974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.754254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.754529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.755362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:42.755665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:42.757201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:42.793036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 496/973] 10.222.0.27 () {56 vars in 1101 bytes} [Thu Jul 14 12:00:42 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:42.793794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.833992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:42.884915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b710ab27-2054-4fbf-ae71-9da49eca1ce7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.939997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.941970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.942419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.942875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.943658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11724ab9-81e6-452d-9b6a-06551b170f31 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:42.943969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:42.945920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:42.966759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:42.968031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:42.969046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:42.971957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 478/974] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:42 2022] GET /identity/v3/services/compute => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:42.978858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:42.991428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:42.993245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:42.993547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:42.993825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:42.994590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:42.994948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:42.996832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.007696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 497/975] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Jul 14 12:00:42 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:43.013242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.066663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:43.108649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6a45de-99cb-4e5c-a4f0-50d691f0aa02 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.137081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a840b402-fa22-4466-a0a2-3d49918f0c9d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.146749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.147442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.147726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.148003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.148705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:43.149010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.150629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.152448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:43.156361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 479/976] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:42 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:43.162601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.175810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.176598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.177008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.177296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.188279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.189758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:43.207686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:43.208583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:43.209668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:43.209668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:43.209668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:43.209668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:43.209668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 498/977] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:43.214334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.274258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97f58fc7-87b1-4dfe-b082-fa7c3e6f9237 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.317120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:43.323483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.324168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.324512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.324799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.325517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:43.330720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.330720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.330905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f85661c-4569-4591-a25b-785de8349c05 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.339603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 480/978] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:43.340658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.368213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.369363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.369652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.369927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.370651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:43.370982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.372345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.385607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 499/979] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:43.390958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.443182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:43.490542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4a7fee-db58-41b2-9003-24c35c1676a2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.503670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4180604-b7d3-4c97-8255-2c6e51315fdc None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.531495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.533189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.534190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.535335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.536012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] RBAC: Adding query filter params (interface=public, service_id=a81964bc3f7044f8a3261e7dc3586789, region_id=RegionOne) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:43.536336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.537721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.546814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 481/980] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/endpoints?service_id=a81964bc3f7044f8a3261e7dc3586789&interface=public®ion_id=RegionOne => generated 194 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:43.549287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.554051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.554903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.555189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.555466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.564795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.566257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:43.583235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:43.584204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:43.585430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 500/981] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:43.591662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.603341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:43.678440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d23203-5a09-4dc2-90e1-fa690e6e4c75 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.719303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:43.720402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2539329e-ca6c-4a55-96bf-193242e19411 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.731879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.732544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.732991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.733102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.733792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.735218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.742526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 482/982] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/services => generated 1727 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:00:43.761206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.761908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.762191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.762466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:43.763465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:43.763784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:43.765282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:43.786820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 501/983] 10.222.0.27 () {56 vars in 1149 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:43.792476 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:43.907960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c6fc9-0a32-44e2-a64f-8761f96a2a29 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:43.942573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:43.943263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:43.943545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:43.943822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.002147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:44.011351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.012846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.112743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 483/984] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:43 2022] GET /identity/v3/auth/tokens => generated 1560 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 14 12:00:44.118181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.184389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:44.217890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1d3421-fef9-452b-9b9e-5d1a78d7dfdf None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.254290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efb52640-212e-4797-9b48-dc558fde818a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:44.255112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb52640-212e-4797-9b48-dc558fde818a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.255396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb52640-212e-4797-9b48-dc558fde818a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.255671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efb52640-212e-4797-9b48-dc558fde818a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.265704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb52640-212e-4797-9b48-dc558fde818a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.267708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efb52640-212e-4797-9b48-dc558fde818a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-efb52640-212e-4797-9b48-dc558fde818a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:44.286080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:44.287068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:44.287952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:44.287952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:44.287952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:44.287952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:44.291137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 502/985] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:44.292890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.383190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:44.392490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6610042-498b-4638-9900-a716a757ee01 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.423498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5069e9ad-cb85-4a02-a1ff-69b55a8ceafc None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.423695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5069e9ad-cb85-4a02-a1ff-69b55a8ceafc None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.423957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5069e9ad-cb85-4a02-a1ff-69b55a8ceafc None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.425504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 503/986] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:44.427970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9157332-660f-44b4-a478-846f28638629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:44.428638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9157332-660f-44b4-a478-846f28638629 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.429108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9157332-660f-44b4-a478-846f28638629 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.429527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9157332-660f-44b4-a478-846f28638629 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.430283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9157332-660f-44b4-a478-846f28638629 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:44.430594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9157332-660f-44b4-a478-846f28638629 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.430700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd5856f-e997-41bd-88f1-91e93ab15f81 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.431012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd5856f-e997-41bd-88f1-91e93ab15f81 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.431285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcd5856f-e997-41bd-88f1-91e93ab15f81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.432026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9157332-660f-44b4-a478-846f28638629 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.450228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 484/987] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:44.450402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-fcd5856f-e997-41bd-88f1-91e93ab15f81 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:44.450402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fcd5856f-e997-41bd-88f1-91e93ab15f81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:44.451162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9157332-660f-44b4-a478-846f28638629 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.522609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 504/988] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:44 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:44.546373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd5856f-e997-41bd-88f1-91e93ab15f81 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.567194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9157332-660f-44b4-a478-846f28638629 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.595952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:44.605472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:44.606251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.606630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.606938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.614221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.615695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:44.631230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:44.632307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:44.633490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:44.637299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 485/989] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:44.638132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.653495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcd5856f-e997-41bd-88f1-91e93ab15f81 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.654544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:44.691267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:44.692068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.692353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.692627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.693277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.694644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:44.707411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:44.708421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:44.709896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 505/990] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/services/object-store => generated 93 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:44.714420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.767702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acb7b84-00cf-429d-9a53-906d7a85df84 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.805324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:44.805989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.806319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.806598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.807854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:44.808329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.809763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.820443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6755cd0-81d0-47bc-950e-23bcfd816908 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.822293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 486/991] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:44.827946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.835078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:44.857511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:44.858339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.858714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.859040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.859750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:44.860815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.861956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.867900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 506/992] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:44.873217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:44.908485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:44.941193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4195e772-476d-45c0-80ee-bc34c16bc4a2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.976744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:44.977323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:44.977599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:44.977871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:44.978689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] RBAC: Adding query filter params (role.id=bc3dc6d5822c488dafecfc5fbf99ef8b, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=1843ddb84e8b434dbed5ebd32419b8d6) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:44.979041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:44.980464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:44.984211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c9a87b-3f6e-4fdb-a3b1-afd0d9bbb881 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:44.995247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 487/993] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/role_assignments?user.id=1843ddb84e8b434dbed5ebd32419b8d6&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=bc3dc6d5822c488dafecfc5fbf99ef8b => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:45.022650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:45.023468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:45.023823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:45.024220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:45.025282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:45.025661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:45.027168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:45.033816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 507/994] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:44 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:45.039863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:45.151948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ea5b44-f352-4aeb-a0b3-5ab59923c801 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:45.191007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4fec9a7-2ba9-4960-9712-f30c4859b952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:45.191682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fec9a7-2ba9-4960-9712-f30c4859b952 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:45.191975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fec9a7-2ba9-4960-9712-f30c4859b952 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:45.192831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4fec9a7-2ba9-4960-9712-f30c4859b952 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:45.193052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fec9a7-2ba9-4960-9712-f30c4859b952 None admin] RBAC: Adding query filter params (interface=admin, service_id=5c738d39141847b8b8b0b025cbb8ed76, region_id=RegionOne) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:45.193368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fec9a7-2ba9-4960-9712-f30c4859b952 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:45.194732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4fec9a7-2ba9-4960-9712-f30c4859b952 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:45.197621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:45.200888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 488/995] 10.222.0.27 () {56 vars in 1249 bytes} [Thu Jul 14 12:00:45 2022] GET /identity/v3/endpoints?service_id=5c738d39141847b8b8b0b025cbb8ed76&interface=admin®ion_id=RegionOne => generated 193 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:45.207073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:45.252520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:45.311788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284a08c5-5eb5-47b4-88f8-c4bd7e774bb6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:45.345730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca1f7992-e83a-4e6e-82c5-7dfb0baed4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:45.346568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1f7992-e83a-4e6e-82c5-7dfb0baed4d5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:45.346878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1f7992-e83a-4e6e-82c5-7dfb0baed4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:45.347160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1f7992-e83a-4e6e-82c5-7dfb0baed4d5 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:45.347842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1f7992-e83a-4e6e-82c5-7dfb0baed4d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:45.349214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca1f7992-e83a-4e6e-82c5-7dfb0baed4d5 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:45.362864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 508/996] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:45 2022] GET /identity/v3/services => generated 1727 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:00:45.843324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c69c4-e8bb-4683-866a-d0f5da001665 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:45.843847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c69c4-e8bb-4683-866a-d0f5da001665 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:45.843957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a82c69c4-e8bb-4683-866a-d0f5da001665 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:45.845874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 489/997] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:45.851301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5036824-d3ee-4cf2-ab02-8c7da4ca7322 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:45.851702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5036824-d3ee-4cf2-ab02-8c7da4ca7322 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:45.851867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5036824-d3ee-4cf2-ab02-8c7da4ca7322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:45.869394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c5036824-d3ee-4cf2-ab02-8c7da4ca7322 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:45.870586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c5036824-d3ee-4cf2-ab02-8c7da4ca7322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:45.941520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 509/998] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:45 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:45.963994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82c69c4-e8bb-4683-866a-d0f5da001665 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:46.070061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a82c69c4-e8bb-4683-866a-d0f5da001665 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:46.104066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:46.104918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:46.105203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:46.105483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:46.106194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:46.107647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:46.119873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:46.120890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 14 12:00:46.121711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:46.128998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 490/999] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:45 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:46.129191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5036824-d3ee-4cf2-ab02-8c7da4ca7322 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:46.164899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:46.232337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5036824-d3ee-4cf2-ab02-8c7da4ca7322 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:46.265266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:46.265920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:46.266200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:46.266471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:46.267744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:46.268053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:46.269421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:46.274403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 510/1000] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:46 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:46.280197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:46.342398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:46.382249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb1a91c-f07b-4d90-89f0-51cf95e9d542 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:46.415685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ea6627-a7a4-443d-a6a1-abeeb9b0af06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:46.416347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea6627-a7a4-443d-a6a1-abeeb9b0af06 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:46.416630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea6627-a7a4-443d-a6a1-abeeb9b0af06 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:46.417043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ea6627-a7a4-443d-a6a1-abeeb9b0af06 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:46.417751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea6627-a7a4-443d-a6a1-abeeb9b0af06 None admin] RBAC: Adding query filter params (type=compute) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:46.418058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea6627-a7a4-443d-a6a1-abeeb9b0af06 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:46.419451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ea6627-a7a4-443d-a6a1-abeeb9b0af06 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:46.425032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 491/1001] 10.222.0.27 () {56 vars in 1112 bytes} [Thu Jul 14 12:00:46 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:46.430907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:46.535697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:46.539705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c182a6d6-8e1b-4ef4-97e5-921864a41025 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:46.577076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517927ee-b0ee-467e-8705-00434b722024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:46.577742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517927ee-b0ee-467e-8705-00434b722024 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:46.578028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517927ee-b0ee-467e-8705-00434b722024 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:46.578304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517927ee-b0ee-467e-8705-00434b722024 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:46.579103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517927ee-b0ee-467e-8705-00434b722024 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:46.580472 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517927ee-b0ee-467e-8705-00434b722024 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:46.676814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 511/1002] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:46 2022] POST /identity/v3/endpoints => generated 334 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:47.063007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec5aa1-eb88-4ae2-9552-1de0f1261e2f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.063634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec5aa1-eb88-4ae2-9552-1de0f1261e2f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.063634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec5aa1-eb88-4ae2-9552-1de0f1261e2f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.065330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 492/1003] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:47.070268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c59395-f94c-4576-9ce9-89eadc46fdac None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.070678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c59395-f94c-4576-9ce9-89eadc46fdac None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.070892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c59395-f94c-4576-9ce9-89eadc46fdac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.088860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f6c59395-f94c-4576-9ce9-89eadc46fdac None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:47.090045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f6c59395-f94c-4576-9ce9-89eadc46fdac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:47.165136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 512/1004] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:47 2022] POST /identity/v3/auth/tokens => generated 1716 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:47.186637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ec5aa1-eb88-4ae2-9552-1de0f1261e2f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:47.288739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ec5aa1-eb88-4ae2-9552-1de0f1261e2f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:47.314795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:47.322901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222be078-e819-4551-90b6-66c7550511aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:47.330063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222be078-e819-4551-90b6-66c7550511aa None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.330063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222be078-e819-4551-90b6-66c7550511aa None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.330063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222be078-e819-4551-90b6-66c7550511aa None admin] PATH_INFO: `/v3/services/image` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.330063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222be078-e819-4551-90b6-66c7550511aa None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:47.330063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222be078-e819-4551-90b6-66c7550511aa None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-222be078-e819-4551-90b6-66c7550511aa None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:47.338401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:47.339381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:47.340241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:47.341658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 493/1005] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Jul 14 12:00:47 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:47.345933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c59395-f94c-4576-9ce9-89eadc46fdac None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:47.409167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f6b1c-a0ce-4ad0-8744-0f1b702bfc17 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.409444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f6b1c-a0ce-4ad0-8744-0f1b702bfc17 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.409722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-484f6b1c-a0ce-4ad0-8744-0f1b702bfc17 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.411119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 494/1006] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:47.416111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d8d8d6-8795-4201-8249-ebc0cb1de90b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.416399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d8d8d6-8795-4201-8249-ebc0cb1de90b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.416678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d8d8d6-8795-4201-8249-ebc0cb1de90b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.429650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:47.433755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-62d8d8d6-8795-4201-8249-ebc0cb1de90b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:47.434980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-62d8d8d6-8795-4201-8249-ebc0cb1de90b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:47.451673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c59395-f94c-4576-9ce9-89eadc46fdac None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:47.488518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:47.489195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.489483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.489764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.490478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] RBAC: Adding query filter params (name=image) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:47.490809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:47.492508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:47.498865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 513/1007] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:47 2022] GET /identity/v3/services?name=image => generated 122 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:47.504094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:47.512900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 495/1008] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:47 2022] POST /identity/v3/auth/tokens => generated 1716 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:47.536779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d8d8d6-8795-4201-8249-ebc0cb1de90b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:47.613613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ff4d1-911e-469f-9bcf-716f75834d62 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:47.649295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:47.650211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.650493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.650792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.651483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] RBAC: Adding query filter params (type=image) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:47.651790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:47.652027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d8d8d6-8795-4201-8249-ebc0cb1de90b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:47.653158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:47.658382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 514/1009] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:47 2022] GET /identity/v3/services?type=image => generated 122 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:47.678629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:47.690741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:47.691617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.691952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.692175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.692858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:47.694250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:47.708261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:47.709339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:47.715627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 496/1010] 10.222.0.27 () {56 vars in 1126 bytes} [Thu Jul 14 12:00:47 2022] GET /identity/v3/services/object-store => generated 93 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:47.717827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:47.730697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:47.825996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ce283f2-64e6-4aa5-9cdc-4e962f366b3a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:47.861793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:47.862459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:47.862740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:47.863062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:47.863824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:47.864082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:47.865471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:47.870655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 515/1011] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:47 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:47.877038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:47.982453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7accea-1fd6-47e9-ab04-75b88cacec2e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:48.016435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40993e2-d322-4fde-b9a1-9b1a4678861d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:48.017700 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40993e2-d322-4fde-b9a1-9b1a4678861d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:48.018049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40993e2-d322-4fde-b9a1-9b1a4678861d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:48.018390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40993e2-d322-4fde-b9a1-9b1a4678861d None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:48.019182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40993e2-d322-4fde-b9a1-9b1a4678861d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:48.019555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40993e2-d322-4fde-b9a1-9b1a4678861d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:48.020975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40993e2-d322-4fde-b9a1-9b1a4678861d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:48.030699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 497/1012] 10.222.0.27 () {56 vars in 1122 bytes} [Thu Jul 14 12:00:47 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:48.031955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:48.103047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:48.132897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04bdbe32-7ade-4e63-b73a-23e8b5f14cac None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:48.166975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf8f24c-032b-4b71-92f1-2db2a169cd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:48.167634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf8f24c-032b-4b71-92f1-2db2a169cd7d None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:48.167915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf8f24c-032b-4b71-92f1-2db2a169cd7d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:48.168190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf8f24c-032b-4b71-92f1-2db2a169cd7d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:48.168933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf8f24c-032b-4b71-92f1-2db2a169cd7d None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:48.170297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf8f24c-032b-4b71-92f1-2db2a169cd7d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:48.238811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 516/1013] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:48 2022] POST /identity/v3/endpoints => generated 325 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:00:48.845403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04f1e-a23f-4e59-a9c8-1b4683afb380 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:48.845929 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04f1e-a23f-4e59-a9c8-1b4683afb380 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:48.846007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c04f1e-a23f-4e59-a9c8-1b4683afb380 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:48.847524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 498/1014] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:48.852665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf330f-084e-46b7-bb52-8e38e33ae228 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:48.852960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf330f-084e-46b7-bb52-8e38e33ae228 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:48.853238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acf330f-084e-46b7-bb52-8e38e33ae228 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:48.870203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9acf330f-084e-46b7-bb52-8e38e33ae228 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:48.871430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9acf330f-084e-46b7-bb52-8e38e33ae228 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:48.947421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 517/1015] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:48 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:48.971418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04f1e-a23f-4e59-a9c8-1b4683afb380 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:49.024011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:49.075806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c04f1e-a23f-4e59-a9c8-1b4683afb380 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:49.109091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da432316-3304-468b-b168-86632a894dbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:49.109895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da432316-3304-468b-b168-86632a894dbf None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:49.110202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da432316-3304-468b-b168-86632a894dbf None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:49.110488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da432316-3304-468b-b168-86632a894dbf None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:49.120711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da432316-3304-468b-b168-86632a894dbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-da432316-3304-468b-b168-86632a894dbf None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:49.129745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:49.130659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:49.131799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:49.135952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 499/1016] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Jul 14 12:00:48 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:49.137304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acf330f-084e-46b7-bb52-8e38e33ae228 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:49.197579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:49.240655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acf330f-084e-46b7-bb52-8e38e33ae228 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:49.276256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:49.277223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:49.277704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:49.278211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:49.279224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:49.279736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:49.281298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:49.294502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 518/1017] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Jul 14 12:00:49 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:49.296276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da432316-3304-468b-b168-86632a894dbf None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:49.401609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da432316-3304-468b-b168-86632a894dbf None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:49.412564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:49.436258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:49.437040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:49.437320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:49.437593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:49.447886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:49.449320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:49.466724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:49.467661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:49.468588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:49.468588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:49.468588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:49.468588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:49.468588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 500/1018] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:49.473960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:49.575860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7567d1c4-1238-4be6-89f9-b9fba9294ddc None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:49.623886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:49.638597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b5c96d-0fb5-41d0-b3a0-b436d80b8f81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:49.639546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b5c96d-0fb5-41d0-b3a0-b436d80b8f81 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:49.640034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b5c96d-0fb5-41d0-b3a0-b436d80b8f81 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:49.640617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b5c96d-0fb5-41d0-b3a0-b436d80b8f81 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:49.641604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b5c96d-0fb5-41d0-b3a0-b436d80b8f81 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:49.642097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b5c96d-0fb5-41d0-b3a0-b436d80b8f81 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:49.643805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3b5c96d-0fb5-41d0-b3a0-b436d80b8f81 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:49.661597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 519/1019] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:49.663203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:49.727947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80963963-354a-44eb-b81a-9093c8b23c3b None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:49.727947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80963963-354a-44eb-b81a-9093c8b23c3b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:49.728127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80963963-354a-44eb-b81a-9093c8b23c3b None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:49.729560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 520/1020] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:49.734811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910df148-eb89-4286-9420-f7aace68d9eb None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:49.735105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910df148-eb89-4286-9420-f7aace68d9eb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:49.735955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910df148-eb89-4286-9420-f7aace68d9eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:49.752658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-910df148-eb89-4286-9420-f7aace68d9eb None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:49.753848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-910df148-eb89-4286-9420-f7aace68d9eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:49.788007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3eb4ef4-c8bf-4f0e-8bc9-0329119e31f6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:49.823657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:49.824483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:49.824724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:49.825062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:49.826631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 521/1021] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:49 2022] POST /identity/v3/auth/tokens => generated 1863 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:49.833000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:49.834450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:49.849199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:49.855263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:49.856423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:49.857081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910df148-eb89-4286-9420-f7aace68d9eb None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:49.857169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 501/1022] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 12:00:49 2022] GET /identity/v3/users/nova => generated 82 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:49.858587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:49.887860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:49.962798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910df148-eb89-4286-9420-f7aace68d9eb None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:49.970108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfc2dbb6-20a9-410f-8c96-fc68fdc32a40 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:49.975130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:50.000085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935d01ba-bdb2-443e-a57a-c1ee5a2fdac6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.001239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935d01ba-bdb2-443e-a57a-c1ee5a2fdac6 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.001697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935d01ba-bdb2-443e-a57a-c1ee5a2fdac6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.002380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935d01ba-bdb2-443e-a57a-c1ee5a2fdac6 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.003146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935d01ba-bdb2-443e-a57a-c1ee5a2fdac6 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.004521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935d01ba-bdb2-443e-a57a-c1ee5a2fdac6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.008903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.009573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.009857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.010131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.011003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:50.011317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.012737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.031508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 502/1023] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 12:00:49 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:50.037908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:50.074185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 522/1024] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:49 2022] POST /identity/v3/services => generated 242 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:00:50.128634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:50.151496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d45e483-326b-4872-9811-9dbaad4d48b7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:50.197220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.197879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.198160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.198435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.271660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.273114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.280327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468fdaf1-7311-4ce4-b98d-94e520b34e90 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.280808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468fdaf1-7311-4ce4-b98d-94e520b34e90 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.280886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-468fdaf1-7311-4ce4-b98d-94e520b34e90 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.282918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 523/1025] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:50.287880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1968fd4-f739-41b0-b353-d6e35f664616 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.288169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1968fd4-f739-41b0-b353-d6e35f664616 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.288445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1968fd4-f739-41b0-b353-d6e35f664616 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.304652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a1968fd4-f739-41b0-b353-d6e35f664616 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:50.305858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a1968fd4-f739-41b0-b353-d6e35f664616 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:50.355519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:50.379050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 524/1026] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:50 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:50.380759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 503/1027] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:50 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 14 12:00:50.388317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1968fd4-f739-41b0-b353-d6e35f664616 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:50.403334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:50.485742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:50.488707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1968fd4-f739-41b0-b353-d6e35f664616 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:50.512855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16124414-6f9f-459b-a468-1b24de87637a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:50.525261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.526401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.526909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.527540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.543176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.544645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.551995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.552442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.552729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.553007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.554179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.555637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.557545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 504/1028] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:50.562854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:50.564378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:50.565471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:50.565471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:50.565471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:50.565471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:50.565693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:50.565767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 525/1029] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:50.570800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:50.618634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:50.680142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d06d7b-2c75-40e2-85ca-39c59e9abff7 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:50.680355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123a54a0-fb8b-431f-a4fb-3693577f66e2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:50.715040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.715710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.715995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.716276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.716990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.718426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.720084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.721095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.721520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.721943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.725771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:50.726265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.728260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.741406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 526/1030] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:50.753505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:50.762634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:50.808607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 505/1031] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:00:50 2022] POST /identity/v3/projects => generated 311 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:00:50.856682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4d507b-e543-4323-8dfd-a5ad7476cd8f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:50.894712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:50.895553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:50.895834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:50.896110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:50.903998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:50.905475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:50.921192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:50.922186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:50.924408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 527/1032] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:50.930002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:51.032448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:51.038640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9da6c096-3bc7-49df-bb92-aa2782627058 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:51.073547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509bf530-760a-402a-9433-44442032a99d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:51.074248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509bf530-760a-402a-9433-44442032a99d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:51.074533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509bf530-760a-402a-9433-44442032a99d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:51.074841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509bf530-760a-402a-9433-44442032a99d None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:51.075673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509bf530-760a-402a-9433-44442032a99d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:51.075985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509bf530-760a-402a-9433-44442032a99d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:51.077419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509bf530-760a-402a-9433-44442032a99d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:51.089607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 506/1033] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:51.096616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:51.170542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:51.202558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014c66c2-1db8-4797-b149-19a3746b1fc0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:51.243311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b94b85-a8bc-4c90-83e8-f7cd5df6cb8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:51.243976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b94b85-a8bc-4c90-83e8-f7cd5df6cb8a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:51.244354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b94b85-a8bc-4c90-83e8-f7cd5df6cb8a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:51.244644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b94b85-a8bc-4c90-83e8-f7cd5df6cb8a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:51.245489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b94b85-a8bc-4c90-83e8-f7cd5df6cb8a None admin] RBAC: Adding query filter params (role.id=bc3dc6d5822c488dafecfc5fbf99ef8b, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=da24a7ecdcc249f0820add0404a7856c) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:51.245803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b94b85-a8bc-4c90-83e8-f7cd5df6cb8a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:51.247323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39b94b85-a8bc-4c90-83e8-f7cd5df6cb8a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:51.253380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 528/1034] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:51 2022] GET /identity/v3/role_assignments?user.id=da24a7ecdcc249f0820add0404a7856c&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=bc3dc6d5822c488dafecfc5fbf99ef8b => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:52.147884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee73f2a-3ea9-4593-8942-ee4cf8a2636b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.148405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee73f2a-3ea9-4593-8942-ee4cf8a2636b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.148405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ee73f2a-3ea9-4593-8942-ee4cf8a2636b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.149868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 507/1035] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:52.154967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8849b070-923d-41c8-a6b8-21102bc61692 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.155365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8849b070-923d-41c8-a6b8-21102bc61692 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.155536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8849b070-923d-41c8-a6b8-21102bc61692 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.172272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8849b070-923d-41c8-a6b8-21102bc61692 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:52.174084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8849b070-923d-41c8-a6b8-21102bc61692 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:52.241047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 529/1036] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:52 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:52.263353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee73f2a-3ea9-4593-8942-ee4cf8a2636b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:52.367300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ee73f2a-3ea9-4593-8942-ee4cf8a2636b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:52.402241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:52.402966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.403245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.403514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] PATH_INFO: `/v3/services/image` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.404157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:52.405511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:52.417503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:52.418549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:52.419471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:52.423247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 508/1037] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Jul 14 12:00:52 2022] GET /identity/v3/services/image => generated 86 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:52.424624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8849b070-923d-41c8-a6b8-21102bc61692 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:52.489826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:52.523462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8849b070-923d-41c8-a6b8-21102bc61692 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:52.558344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:52.559028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.559306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.559578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.560311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] RBAC: Adding query filter params (name=image) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:52.560578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:52.561938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:52.572386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 530/1038] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:52 2022] GET /identity/v3/services?name=image => generated 122 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:52.573984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:52.597828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:52.676836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c352896b-17de-4ad3-9e3e-0c714a5bab02 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:52.713715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:52.713715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.713715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.713715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.714120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] RBAC: Adding query filter params (type=image) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:52.714120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:52.714537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:52.720984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 509/1039] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:52 2022] GET /identity/v3/services?type=image => generated 350 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:52.726742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:52.829304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b8a046-48eb-400d-ace4-a08d18dc966b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:52.858116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:52.865087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36c553d-7c73-47bc-8712-9078c102eb90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:52.865817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36c553d-7c73-47bc-8712-9078c102eb90 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.866150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36c553d-7c73-47bc-8712-9078c102eb90 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.871062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36c553d-7c73-47bc-8712-9078c102eb90 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.871062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36c553d-7c73-47bc-8712-9078c102eb90 None admin] RBAC: Adding query filter params (interface=public, service_id=95b300104c37421cbc7ab3cadc9a8ffa, region_id=RegionOne) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:52.871062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36c553d-7c73-47bc-8712-9078c102eb90 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:52.871062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36c553d-7c73-47bc-8712-9078c102eb90 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:52.879152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 531/1040] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:00:52 2022] GET /identity/v3/endpoints?service_id=95b300104c37421cbc7ab3cadc9a8ffa&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:52.880358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:52.934021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1840c905-2436-4a3b-81de-0113a5f82163 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.934650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1840c905-2436-4a3b-81de-0113a5f82163 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.934962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1840c905-2436-4a3b-81de-0113a5f82163 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.936358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 532/1041] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:52.941718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c063433-9a53-4655-be4f-a66cb52e2178 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:52.942001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c063433-9a53-4655-be4f-a66cb52e2178 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:52.942280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c063433-9a53-4655-be4f-a66cb52e2178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:52.959369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-7c063433-9a53-4655-be4f-a66cb52e2178 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:52.960671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7c063433-9a53-4655-be4f-a66cb52e2178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:52.974001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:52.990053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28073384-40ea-4b43-86c8-aee3f1b89b5a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:53.027172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56707726-fcb8-4791-b210-938168170492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:53.027838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56707726-fcb8-4791-b210-938168170492 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.028127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56707726-fcb8-4791-b210-938168170492 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.028401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56707726-fcb8-4791-b210-938168170492 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.029187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56707726-fcb8-4791-b210-938168170492 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:53.030540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56707726-fcb8-4791-b210-938168170492 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:53.037937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 510/1042] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:00:52 2022] GET /identity/v3/services => generated 1957 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:00:53.048726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 533/1043] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:52 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:53.074745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56707726-fcb8-4791-b210-938168170492 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:53.186456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56707726-fcb8-4791-b210-938168170492 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:53.223136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34908b2-d5c5-4862-90fd-b0da784b1a63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:53.223931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34908b2-d5c5-4862-90fd-b0da784b1a63 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.224222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34908b2-d5c5-4862-90fd-b0da784b1a63 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.224495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34908b2-d5c5-4862-90fd-b0da784b1a63 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.225679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34908b2-d5c5-4862-90fd-b0da784b1a63 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:53.227140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34908b2-d5c5-4862-90fd-b0da784b1a63 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:53.228932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 511/1044] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:53.234873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c063433-9a53-4655-be4f-a66cb52e2178 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:53.284184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb91ef0-5282-40c8-b475-77900bf9a983 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.284557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb91ef0-5282-40c8-b475-77900bf9a983 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.284738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecb91ef0-5282-40c8-b475-77900bf9a983 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.286244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 512/1045] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:53.291441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5493c2-5f80-40dd-b69d-9ca0ef6efa7c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.291809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5493c2-5f80-40dd-b69d-9ca0ef6efa7c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.292423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5493c2-5f80-40dd-b69d-9ca0ef6efa7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.314815 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2b5493c2-5f80-40dd-b69d-9ca0ef6efa7c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:53.316480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2b5493c2-5f80-40dd-b69d-9ca0ef6efa7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:53.328699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:53.341897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c063433-9a53-4655-be4f-a66cb52e2178 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:53.369475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:53.381273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:53.381992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.382280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.382816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.383300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:53.384740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:53.387545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 513/1046] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:53 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:53.410999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5493c2-5f80-40dd-b69d-9ca0ef6efa7c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:53.493920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 534/1047] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 12:00:53 2022] POST /identity/v3/users => generated 285 bytes in 259 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:00:53.518486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5493c2-5f80-40dd-b69d-9ca0ef6efa7c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:53.561243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:53.562159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.562444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.562721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.575424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:53.584016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:53.585196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:53.586241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:53.590463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 514/1048] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Jul 14 12:00:53 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:53.591925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:53.704284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390b6bcb-119e-4f35-b079-4268501fd36d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:53.745224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:53.746121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.746556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.747050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.747965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:53.748436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:53.749965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:53.758717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 535/1049] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Jul 14 12:00:53 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:53.763947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:53.774468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:53.867900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a82cb5-79fa-4d24-81b8-04a9bc8f8060 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:53.902106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:53.902915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:53.903195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:53.903463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:53.914316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:53.915796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:53.933432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:53.934372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:53.935365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:53.935365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:53.935365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:53.935365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:53.935365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 515/1050] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:53.940983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:53.984239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:54.041907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a1b61a-8a9c-48c3-a169-68e14b0cf621 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:54.076477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:54.077134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:54.077409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:54.077678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:54.078386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:54.078688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:54.080508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:54.097300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 536/1051] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:54.098566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:54.158151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:54.203169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93e5653-1683-4de0-8551-dab7cb810c48 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:54.238403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:54.239243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:54.239530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:54.239813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] PATH_INFO: `/v3/users/nova` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:54.247456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:54.248907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:54.263547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:54.264545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:54.265492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 516/1052] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 12:00:54 2022] GET /identity/v3/users/nova => generated 82 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:54.270930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:54.371002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c154bbb0-f7b3-4ba9-aa9e-fcbda6b4672c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:54.372444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:54.404401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:54.405131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:54.405462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:54.405795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:54.406573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:54.407440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:54.408875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:54.431235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 537/1053] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 12:00:54 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:54.432693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:54.535854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd9ba3d-86b1-49f6-8d2b-431e626c455e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:54.553278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:54.570440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:54.571266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:54.571616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:54.577188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:54.583114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:54.584564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:54.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:54.603692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:54.604591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:54.604591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:54.604591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:54.604591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:54.604591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 517/1054] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:54.611308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:54.721854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66bacdf-eb61-461d-8b4f-29f0573b434f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:54.760360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:54.761020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:54.761302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:54.761577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:54.762295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:54.762602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:54.764003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:54.775967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 538/1055] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:54.781324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:54.885128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0fc94c-04fe-42f7-a988-96bee469e2aa None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:54.921331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:54.922122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:54.922409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:54.923409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:54.930477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:54.931945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:54.946321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:54.947425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:54.948329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 518/1056] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:54.955064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:54.978865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:55.057263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55da6a0-b54f-441f-9280-5247283c4427 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:55.090745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d53ad9d-d1ce-4428-9087-16dcc77ec388 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:55.091442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53ad9d-d1ce-4428-9087-16dcc77ec388 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.091724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53ad9d-d1ce-4428-9087-16dcc77ec388 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.092001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d53ad9d-d1ce-4428-9087-16dcc77ec388 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.092787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d53ad9d-d1ce-4428-9087-16dcc77ec388 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:55.093098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d53ad9d-d1ce-4428-9087-16dcc77ec388 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:55.095199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d53ad9d-d1ce-4428-9087-16dcc77ec388 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:55.110822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 539/1057] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:55.114001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:55.123692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f55f7d-9c88-4f41-98e4-abfb7f8c657e None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.123989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f55f7d-9c88-4f41-98e4-abfb7f8c657e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.124268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f55f7d-9c88-4f41-98e4-abfb7f8c657e None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.126353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 540/1058] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:55.131820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9385ef-58eb-4ea6-87ce-1502dd429f4d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.132301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9385ef-58eb-4ea6-87ce-1502dd429f4d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.132587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef9385ef-58eb-4ea6-87ce-1502dd429f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.148951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:55.151680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ef9385ef-58eb-4ea6-87ce-1502dd429f4d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:55.152896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ef9385ef-58eb-4ea6-87ce-1502dd429f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:55.226973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 541/1059] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:55 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:55.233025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9798a3cf-1cd8-42c1-95bb-c998401a568e None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:55.249594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9385ef-58eb-4ea6-87ce-1502dd429f4d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:55.261731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:55.272532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d5a273-0001-4971-84a1-249511add80b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:55.273427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d5a273-0001-4971-84a1-249511add80b None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.273712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d5a273-0001-4971-84a1-249511add80b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.273987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d5a273-0001-4971-84a1-249511add80b None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/da24a7ecdcc249f0820add0404a7856c/roles/bc3dc6d5822c488dafecfc5fbf99ef8b` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.277585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d5a273-0001-4971-84a1-249511add80b None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=da24a7ecdcc249f0820add0404a7856c, role_id=bc3dc6d5822c488dafecfc5fbf99ef8b)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:55.279067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d5a273-0001-4971-84a1-249511add80b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:55.354437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef9385ef-58eb-4ea6-87ce-1502dd429f4d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:55.366803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 519/1060] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:55 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/da24a7ecdcc249f0820add0404a7856c/roles/bc3dc6d5822c488dafecfc5fbf99ef8b => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:00:55.404709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:55.405544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.405811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.406080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] PATH_INFO: `/v3/services/image` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.406753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:55.409135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:55.421974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:55.423501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 14 12:00:55.424367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:55.427144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 542/1061] 10.222.0.27 () {56 vars in 1105 bytes} [Thu Jul 14 12:00:55 2022] GET /identity/v3/services/image => generated 86 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:55.429693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d5a273-0001-4971-84a1-249511add80b None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:55.530169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:55.540898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d5a273-0001-4971-84a1-249511add80b None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:55.576336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22ca1b4b-2ccf-4efc-971f-82fed4ac8b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:55.577054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ca1b4b-2ccf-4efc-971f-82fed4ac8b6c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.577343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ca1b4b-2ccf-4efc-971f-82fed4ac8b6c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.577619 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22ca1b4b-2ccf-4efc-971f-82fed4ac8b6c None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.578335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ca1b4b-2ccf-4efc-971f-82fed4ac8b6c None admin] RBAC: Adding query filter params (name=image) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:55.578649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ca1b4b-2ccf-4efc-971f-82fed4ac8b6c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:55.580605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22ca1b4b-2ccf-4efc-971f-82fed4ac8b6c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:55.586100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 520/1062] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:55 2022] GET /identity/v3/services?name=image => generated 122 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:55.591924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:55.615085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208a578-a514-4320-b742-07c1e3e6b261 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.615403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208a578-a514-4320-b742-07c1e3e6b261 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.615647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a208a578-a514-4320-b742-07c1e3e6b261 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.617069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 521/1063] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:55.621910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0592d4f-1a85-45c8-b3d2-130ac868095b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.622195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0592d4f-1a85-45c8-b3d2-130ac868095b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.622650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0592d4f-1a85-45c8-b3d2-130ac868095b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.641834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f0592d4f-1a85-45c8-b3d2-130ac868095b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:55.643965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f0592d4f-1a85-45c8-b3d2-130ac868095b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:55.677193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:55.703918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70106cb-088f-4f0f-8736-51abe8c36c7b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:55.730913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 522/1064] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:55 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:55.746385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:55.747118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.747398 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.747637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0592d4f-1a85-45c8-b3d2-130ac868095b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:55.747743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.748380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] RBAC: Adding query filter params (type=image) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:55.748683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:55.750045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:55.755301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 543/1065] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:55 2022] GET /identity/v3/services?type=image => generated 350 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:55.760874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:55.837282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:55.853967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0592d4f-1a85-45c8-b3d2-130ac868095b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:55.872213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ff437e-a21e-4c39-85d9-196bdd68860a None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:55.890911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:55.891767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.892056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.892336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.903724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:55.908630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:55.909295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:55.909575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:55.909852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:55.910590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:55.911991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:55.912193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:55.913185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:55.914017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:55.918418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 523/1066] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:55.919090 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:55.928223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:56.014043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 544/1067] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:00:55 2022] POST /identity/v3/endpoints => generated 327 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:00:56.034235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548b9606-fcac-4aba-9dc1-f6d04be03dda None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:56.080332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:56.081015 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:56.081293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:56.081561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:56.082320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:56.082625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:56.084055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:56.099775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 524/1068] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:56.101674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:56.213344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2be3c279-5ddc-457b-9f5c-46ef6b1fdd24 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:56.247309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:56.248178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:56.248464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:56.248743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] PATH_INFO: `/v3/users/bf5d2bc18c264ff29bd1d42fe30ea997` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:56.262112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] RBAC: Authorizing `identity:get_user(user_id=bf5d2bc18c264ff29bd1d42fe30ea997)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:56.263587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:56.265432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 545/1069] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:00:56 2022] GET /identity/v3/users/bf5d2bc18c264ff29bd1d42fe30ea997 => generated 285 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:56.271492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:56.274567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:56.373566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2893c92-a1ab-44d7-82b9-bf5ff52bd5c3 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:56.408543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:56.409199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:56.409480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:56.409761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:56.477704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:56.479206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:56.481133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:56.584767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 525/1070] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:56 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:00:56.590594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:56.645330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:56.691718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191bd338-240c-45ee-81f0-dbdbb00790e0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:56.738080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7479a25f-fc0d-4085-9fcf-eff2c85d1101 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:56.738967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479a25f-fc0d-4085-9fcf-eff2c85d1101 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:56.739246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479a25f-fc0d-4085-9fcf-eff2c85d1101 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:56.739517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7479a25f-fc0d-4085-9fcf-eff2c85d1101 None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:56.740681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479a25f-fc0d-4085-9fcf-eff2c85d1101 None admin] RBAC: Authorizing `identity:get_project(project_id=877f67a84bb541eab9460478b8186707)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:56.742106 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7479a25f-fc0d-4085-9fcf-eff2c85d1101 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:56.748727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 546/1071] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:00:56 2022] GET /identity/v3/projects/877f67a84bb541eab9460478b8186707 => generated 311 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:56.750214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:56.850635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87692a8e-d39c-4cac-ab6d-fcb637902618 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:56.852197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:56.884728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-075d9299-eab9-4692-be66-6e5b03a40126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:56.885421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075d9299-eab9-4692-be66-6e5b03a40126 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:56.885704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075d9299-eab9-4692-be66-6e5b03a40126 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:56.885976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-075d9299-eab9-4692-be66-6e5b03a40126 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:56.886859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075d9299-eab9-4692-be66-6e5b03a40126 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=877f67a84bb541eab9460478b8186707, user.id=bf5d2bc18c264ff29bd1d42fe30ea997) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:56.887177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075d9299-eab9-4692-be66-6e5b03a40126 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:56.888605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-075d9299-eab9-4692-be66-6e5b03a40126 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:56.902882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 526/1072] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:56 2022] GET /identity/v3/role_assignments?user.id=bf5d2bc18c264ff29bd1d42fe30ea997&scope.project.id=877f67a84bb541eab9460478b8186707&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:57.434736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db9b38-f18c-490c-839e-07cc8ae36e5b None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:57.435028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db9b38-f18c-490c-839e-07cc8ae36e5b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:57.435301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db9b38-f18c-490c-839e-07cc8ae36e5b None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:57.436855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 547/1073] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:00:57.441732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d636fb-dcca-47d3-ab62-d4550c6d60ee None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:57.442175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d636fb-dcca-47d3-ab62-d4550c6d60ee None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:57.442268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d636fb-dcca-47d3-ab62-d4550c6d60ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:57.459333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-21d636fb-dcca-47d3-ab62-d4550c6d60ee None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:57.460516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-21d636fb-dcca-47d3-ab62-d4550c6d60ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:57.530174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 527/1074] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:00:57.553530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1db9b38-f18c-490c-839e-07cc8ae36e5b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:57.659374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1db9b38-f18c-490c-839e-07cc8ae36e5b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:57.681098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:57.695483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:57.696289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:57.696563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:57.696831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:57.707518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:57.715657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:57.716609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 14 12:00:57.717498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:57.725331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 548/1075] 10.222.0.27 () {56 vars in 1120 bytes} [Thu Jul 14 12:00:57 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:57.726602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d636fb-dcca-47d3-ab62-d4550c6d60ee None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:57.819967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:57.826256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d636fb-dcca-47d3-ab62-d4550c6d60ee None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:57.861193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:57.861863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:57.862146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:57.862418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:57.863204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:57.863513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:57.864864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:57.877951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 528/1076] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Jul 14 12:00:57 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:57.879727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:57.984584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b809b31d-bc62-4322-8043-8364f1bb1e0c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.020968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c81eba-194f-4e21-b7e3-c551adb07e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.021976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c81eba-194f-4e21-b7e3-c551adb07e55 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.022292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c81eba-194f-4e21-b7e3-c551adb07e55 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.022571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c81eba-194f-4e21-b7e3-c551adb07e55 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.034394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c81eba-194f-4e21-b7e3-c551adb07e55 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.035870 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c81eba-194f-4e21-b7e3-c551adb07e55 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-84c81eba-194f-4e21-b7e3-c551adb07e55 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:58.053294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:58.054246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:58.055163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:58.055163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:58.055163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:58.055163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:58.055163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 549/1077] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:58.065764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.095017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd27f4ee-0c52-4dea-8d37-89f9664d3b43 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.095378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd27f4ee-0c52-4dea-8d37-89f9664d3b43 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.095569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd27f4ee-0c52-4dea-8d37-89f9664d3b43 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.097144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 550/1078] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:58.101716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f366bf-8c8d-4560-8e2f-1f03733ea2a1 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.102008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f366bf-8c8d-4560-8e2f-1f03733ea2a1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.102290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f366bf-8c8d-4560-8e2f-1f03733ea2a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.118715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-20f366bf-8c8d-4560-8e2f-1f03733ea2a1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:58.119936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-20f366bf-8c8d-4560-8e2f-1f03733ea2a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:58.147952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:58.174244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e56928-abaf-460f-a4ba-c75964f0ed8c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.203118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 551/1079] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:58 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:58.208775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.209437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.209720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.209996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.210710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:58.211049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.212408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.214742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f366bf-8c8d-4560-8e2f-1f03733ea2a1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.226447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 529/1080] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:58.232524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.251726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:58.324367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f366bf-8c8d-4560-8e2f-1f03733ea2a1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.341179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75c9720d-fe0f-4f96-b8dc-8ef0dbed6727 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.360107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.360918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.361204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.361527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.382291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.383225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.383510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.383788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] PATH_INFO: `/v3/users/nova` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.391093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:58.393530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.395003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:00:58.410702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:58.411887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:58.412843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 530/1081] 10.222.0.27 () {56 vars in 1093 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:58.417585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.434009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.435504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.502715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:58.529484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8610caa4-1173-43de-b0dc-baaae27c301a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.542810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 552/1082] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 326 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 14 12:00:58.545735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.572717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.573394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.573679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.573957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.574829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:58.575149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.576585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.595769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 531/1083] 10.222.0.27 () {56 vars in 1133 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:58.601781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.614669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:58.655932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f15aa0c-9a48-46eb-b6dc-c99d1c77040e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.696941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.698083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.698083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.698276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.709883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.710334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ada229c-8327-4691-89fa-fc6e1e2614d6 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.713832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:58.734400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:58.735423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:58.736328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:58.736328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:58.736328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:58.736328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:58.736328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 553/1084] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:00:58.741950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.752350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.753060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.753284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.753560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.757042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:58.826625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.828194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.852997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fb18877-8b1d-444b-9fc7-8d06540aa7c4 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:58.868801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:58.888952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5144d7c-3bf2-4af3-9630-6f3526772d2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:58.889697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5144d7c-3bf2-4af3-9630-6f3526772d2d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.889885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5144d7c-3bf2-4af3-9630-6f3526772d2d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.890164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5144d7c-3bf2-4af3-9630-6f3526772d2d None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.891032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5144d7c-3bf2-4af3-9630-6f3526772d2d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:58.891281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5144d7c-3bf2-4af3-9630-6f3526772d2d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:58.893363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5144d7c-3bf2-4af3-9630-6f3526772d2d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:58.914812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 554/1085] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:00:58.919876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf1c7c8-44bd-4e97-b0f4-65711bb3b43c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.920159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf1c7c8-44bd-4e97-b0f4-65711bb3b43c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.920427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf1c7c8-44bd-4e97-b0f4-65711bb3b43c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.921864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 555/1086] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:00:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:00:58.928467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f7c6f6-63c2-44c2-8b35-4802397ef40e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:58.928828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f7c6f6-63c2-44c2-8b35-4802397ef40e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:58.929023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f7c6f6-63c2-44c2-8b35-4802397ef40e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:58.948000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d2f7c6f6-63c2-44c2-8b35-4802397ef40e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:00:58.949243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d2f7c6f6-63c2-44c2-8b35-4802397ef40e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:00:58.950442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 532/1087] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 14 12:00:58.956747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:58.995382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:59.028612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 556/1088] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:00:58 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:00:59.052902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f7c6f6-63c2-44c2-8b35-4802397ef40e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.071541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83a4ea9-9f77-4cd6-97d1-479210005011 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.114222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.115065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.115347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.115623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.127709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.129157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:00:59.149107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:59.150380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:59.151438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:00:59.151438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:00:59.151438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:00:59.151438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:59.151438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 533/1089] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:00:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:59.156350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.172169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f7c6f6-63c2-44c2-8b35-4802397ef40e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.174894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:59.212045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.212833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.213109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.213382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.224250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:00:59.232068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:59.233125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:00:59.233963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 557/1090] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:59.239383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.250466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:59.263353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427c5865-52d4-458b-828f-e41339d11a8a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.298608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.299218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.299500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.299802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.300487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:59.300798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.302157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.313978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 534/1091] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:59.319579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.339036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04239a89-0ab2-4d86-b3d8-dfc14ddcaef5 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.371574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.372238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.372519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.372796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.373545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:59.373854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.375259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.383679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 558/1092] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:59.389277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.416581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:59.417469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf204e-0ddc-4466-9735-661ebfa60015 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.450569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.451381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.451666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.451939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.458474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.459937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:00:59.474237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:00:59.475443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:00:59.476366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 535/1093] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:00:59.481044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.489294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9efec9-5926-405f-9579-578a47ddb27e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.491650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:59.523459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.524335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.524619 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.524895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] PATH_INFO: `/v3/users/bf5d2bc18c264ff29bd1d42fe30ea997` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.526443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] RBAC: Authorizing `identity:get_user(user_id=bf5d2bc18c264ff29bd1d42fe30ea997)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.527919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.529912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 559/1094] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/users/bf5d2bc18c264ff29bd1d42fe30ea997 => generated 285 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:59.535852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.585239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000fa7f8-8f9a-475f-979e-5de5710be537 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.619013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.619578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.619864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.620133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.620913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:59.621219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.622643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.629059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:59.634970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 536/1095] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:59.635734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458b22f5-500b-4d00-8d11-dedd074acf94 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.640948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.670468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.671369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.671639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.671914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.674142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] RBAC: Authorizing `identity:get_project(project_id=877f67a84bb541eab9460478b8186707)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.675628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.677331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 560/1096] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/projects/877f67a84bb541eab9460478b8186707 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:59.683117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:00:59.708210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:00:59.743865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de0d8183-1b65-4b56-98bc-cac94063f69f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.776859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402645c0-6e51-4100-b87d-90bbaf1583be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.777513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402645c0-6e51-4100-b87d-90bbaf1583be None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.777794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402645c0-6e51-4100-b87d-90bbaf1583be None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.778065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402645c0-6e51-4100-b87d-90bbaf1583be None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.778917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402645c0-6e51-4100-b87d-90bbaf1583be None admin] RBAC: Adding query filter params (role.id=bc3dc6d5822c488dafecfc5fbf99ef8b, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=da24a7ecdcc249f0820add0404a7856c) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:00:59.779230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402645c0-6e51-4100-b87d-90bbaf1583be None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.780648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-402645c0-6e51-4100-b87d-90bbaf1583be None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.784428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c47b3e3-7410-4423-990b-1c8f2776bc62 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:00:59.787239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 537/1097] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:00:59 2022] GET /identity/v3/role_assignments?user.id=da24a7ecdcc249f0820add0404a7856c&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=bc3dc6d5822c488dafecfc5fbf99ef8b => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:00:59.820462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0836094-61b8-4273-bd0e-b32f362aa10e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:00:59.821369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0836094-61b8-4273-bd0e-b32f362aa10e None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:00:59.821647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0836094-61b8-4273-bd0e-b32f362aa10e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:00:59.821920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0836094-61b8-4273-bd0e-b32f362aa10e None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707/users/bf5d2bc18c264ff29bd1d42fe30ea997/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:00:59.824012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0836094-61b8-4273-bd0e-b32f362aa10e None admin] RBAC: Authorizing `identity:create_grant(project_id=877f67a84bb541eab9460478b8186707, user_id=bf5d2bc18c264ff29bd1d42fe30ea997, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:00:59.825484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0836094-61b8-4273-bd0e-b32f362aa10e None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:00:59.915693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 561/1098] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:00:59 2022] PUT /identity/v3/projects/877f67a84bb541eab9460478b8186707/users/bf5d2bc18c264ff29bd1d42fe30ea997/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:01:01.921695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c5354-0fe2-4b58-ada1-c09f6120db71 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:01.922223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c5354-0fe2-4b58-ada1-c09f6120db71 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:01.922317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a7c5354-0fe2-4b58-ada1-c09f6120db71 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:01.924508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 538/1099] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:01.929549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1db19de-108d-4d77-ba07-7e1cfb391216 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:01.929929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1db19de-108d-4d77-ba07-7e1cfb391216 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:01.930091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1db19de-108d-4d77-ba07-7e1cfb391216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:01.946301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e1db19de-108d-4d77-ba07-7e1cfb391216 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:01.947508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e1db19de-108d-4d77-ba07-7e1cfb391216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:01.963834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:02.013274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 562/1100] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:01 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:02.037472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c5354-0fe2-4b58-ada1-c09f6120db71 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:02.134939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a7c5354-0fe2-4b58-ada1-c09f6120db71 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:02.163566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:02.168242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:02.169016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:02.169298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:02.169574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:02.179808 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:01:02.187282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:02.188578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:01:02.189835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 539/1101] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:01:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:02.194559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1db19de-108d-4d77-ba07-7e1cfb391216 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:02.292348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1db19de-108d-4d77-ba07-7e1cfb391216 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:02.324884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:02.325542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:02.325824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:02.326099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:02.326864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:02.327176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:02.328551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:02.337067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 563/1102] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:01:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:02.342783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:02.438385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f504ec1-7131-48f3-bbbe-4b78d1b047f9 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:02.472626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:02.473478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:02.473756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:02.474029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] PATH_INFO: `/v3/users/bf5d2bc18c264ff29bd1d42fe30ea997` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:02.475227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] RBAC: Authorizing `identity:get_user(user_id=bf5d2bc18c264ff29bd1d42fe30ea997)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:02.476655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:02.478265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 540/1103] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:02 2022] GET /identity/v3/users/bf5d2bc18c264ff29bd1d42fe30ea997 => generated 285 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:02.483996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:02.501116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:02.582568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f32add-530f-435f-812f-e80dbdd7a65c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:02.693868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:02.694505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:02.694809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:02.695084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:02.764066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:02.765553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:02.780093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:02.861982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 564/1104] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:02 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:02.867894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:02.924719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:02.963146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417a05ad-a01c-4a16-a486-c95c75677a47 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:03.001662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bbd3c5d-c05e-428f-9cbf-6c5f609f6f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:03.002589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbd3c5d-c05e-428f-9cbf-6c5f609f6f35 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:03.003003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbd3c5d-c05e-428f-9cbf-6c5f609f6f35 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:03.003374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bbd3c5d-c05e-428f-9cbf-6c5f609f6f35 None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:03.005065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbd3c5d-c05e-428f-9cbf-6c5f609f6f35 None admin] RBAC: Authorizing `identity:get_project(project_id=877f67a84bb541eab9460478b8186707)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:03.007194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bbd3c5d-c05e-428f-9cbf-6c5f609f6f35 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:03.009037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 541/1105] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:02 2022] GET /identity/v3/projects/877f67a84bb541eab9460478b8186707 => generated 311 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:03.015189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:03.112441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80254fd5-ed31-4935-a87f-3052fcc16767 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:03.125888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:03.145094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6910e698-1979-439b-8000-780da7ae6fcb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:03.145737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6910e698-1979-439b-8000-780da7ae6fcb None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:03.146029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6910e698-1979-439b-8000-780da7ae6fcb None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:03.146306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6910e698-1979-439b-8000-780da7ae6fcb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:03.147157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6910e698-1979-439b-8000-780da7ae6fcb None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=877f67a84bb541eab9460478b8186707, user.id=bf5d2bc18c264ff29bd1d42fe30ea997) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:03.147471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6910e698-1979-439b-8000-780da7ae6fcb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:03.148944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6910e698-1979-439b-8000-780da7ae6fcb None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:03.154885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 565/1106] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:01:03 2022] GET /identity/v3/role_assignments?user.id=bf5d2bc18c264ff29bd1d42fe30ea997&scope.project.id=877f67a84bb541eab9460478b8186707&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:05.193971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa700a6c-63a5-46f5-a6d6-acdb27bebcf1 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:05.194507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa700a6c-63a5-46f5-a6d6-acdb27bebcf1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:05.194507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa700a6c-63a5-46f5-a6d6-acdb27bebcf1 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:05.195936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 542/1107] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:05.201251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a061f-02ff-4925-bc46-da9ab5d35b65 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:05.201672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a061f-02ff-4925-bc46-da9ab5d35b65 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:05.201811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54a061f-02ff-4925-bc46-da9ab5d35b65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:05.218149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d54a061f-02ff-4925-bc46-da9ab5d35b65 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:05.219377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d54a061f-02ff-4925-bc46-da9ab5d35b65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:05.284576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 566/1108] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:05 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:05.310256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa700a6c-63a5-46f5-a6d6-acdb27bebcf1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:05.409467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa700a6c-63a5-46f5-a6d6-acdb27bebcf1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:05.435014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:05.442510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e62022-7a43-4f0d-9345-7468a83a41d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:05.443318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e62022-7a43-4f0d-9345-7468a83a41d7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:05.443608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e62022-7a43-4f0d-9345-7468a83a41d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:05.443884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e62022-7a43-4f0d-9345-7468a83a41d7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:05.445089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e62022-7a43-4f0d-9345-7468a83a41d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:05.446513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e62022-7a43-4f0d-9345-7468a83a41d7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:05.448149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 543/1109] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:01:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:05.454078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a061f-02ff-4925-bc46-da9ab5d35b65 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:05.549174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54a061f-02ff-4925-bc46-da9ab5d35b65 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:05.558022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:05.581453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180c36fe-6bdc-4efd-ac9d-7880409fba67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:05.582106 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180c36fe-6bdc-4efd-ac9d-7880409fba67 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:05.582384 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180c36fe-6bdc-4efd-ac9d-7880409fba67 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:05.582656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180c36fe-6bdc-4efd-ac9d-7880409fba67 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:05.583385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180c36fe-6bdc-4efd-ac9d-7880409fba67 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:05.584794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-180c36fe-6bdc-4efd-ac9d-7880409fba67 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:05.675959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 567/1110] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 12:01:05 2022] POST /identity/v3/users => generated 286 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:01:07.757799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adeeeb7-f6ed-43e3-9238-e7a90b8be2c7 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:07.758361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adeeeb7-f6ed-43e3-9238-e7a90b8be2c7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:07.758361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3adeeeb7-f6ed-43e3-9238-e7a90b8be2c7 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:07.759743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 544/1111] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:07.764923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d927ad1-0b8c-43bf-bf13-be26db2afbae None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:07.765407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d927ad1-0b8c-43bf-bf13-be26db2afbae None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:07.765938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d927ad1-0b8c-43bf-bf13-be26db2afbae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:07.784965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3d927ad1-0b8c-43bf-bf13-be26db2afbae None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:07.786163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3d927ad1-0b8c-43bf-bf13-be26db2afbae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:07.853189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 568/1112] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:07 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:07.879001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adeeeb7-f6ed-43e3-9238-e7a90b8be2c7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:07.987118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3adeeeb7-f6ed-43e3-9238-e7a90b8be2c7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:08.024459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:08.025551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:08.025826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:08.026098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] PATH_INFO: `/v3/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:08.027684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] RBAC: Authorizing `identity:get_role(role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:08.029047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:08.030513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 545/1113] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:07 2022] GET /identity/v3/roles/1ff35caa6d284df39cd287349f66c567 => generated 223 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:08.037589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d927ad1-0b8c-43bf-bf13-be26db2afbae None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:08.127685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:08.134234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d927ad1-0b8c-43bf-bf13-be26db2afbae None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:08.169026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:08.169918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:08.170199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:08.170473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] PATH_INFO: `/v3/users/6b50481f1a1e4adbaed7404fddd75280` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:08.183992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] RBAC: Authorizing `identity:get_user(user_id=6b50481f1a1e4adbaed7404fddd75280)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:08.185446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:08.187180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 569/1114] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:08 2022] GET /identity/v3/users/6b50481f1a1e4adbaed7404fddd75280 => generated 286 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:08.193625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:08.256304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:08.294894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5041c8f-cff9-49b8-a062-669008c74cd1 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:08.330132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:08.330807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:08.331093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:08.331370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:08.398603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:08.400082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:08.458136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:08.496821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 546/1115] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:08 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:08.503385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:08.601414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ba9d6d8-9b72-4a4a-bb95-695aa80e3e12 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:08.635172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb941489-4bff-4252-9b66-7e06192a0598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:08.636016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb941489-4bff-4252-9b66-7e06192a0598 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:08.636300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb941489-4bff-4252-9b66-7e06192a0598 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:08.636576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb941489-4bff-4252-9b66-7e06192a0598 None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:08.637752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb941489-4bff-4252-9b66-7e06192a0598 None admin] RBAC: Authorizing `identity:get_project(project_id=877f67a84bb541eab9460478b8186707)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:08.639253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb941489-4bff-4252-9b66-7e06192a0598 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:08.641564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 570/1116] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:08 2022] GET /identity/v3/projects/877f67a84bb541eab9460478b8186707 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:08.647668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:08.758048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b4ee2f-0029-49fc-8795-360e13a31e43 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:08.792976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739908a6-f8f2-49f7-9a7c-86f4dbbc803f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:08.793622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739908a6-f8f2-49f7-9a7c-86f4dbbc803f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:08.793902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739908a6-f8f2-49f7-9a7c-86f4dbbc803f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:08.794175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739908a6-f8f2-49f7-9a7c-86f4dbbc803f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:08.795023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739908a6-f8f2-49f7-9a7c-86f4dbbc803f None admin] RBAC: Adding query filter params (role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=877f67a84bb541eab9460478b8186707, user.id=6b50481f1a1e4adbaed7404fddd75280) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:08.795336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739908a6-f8f2-49f7-9a7c-86f4dbbc803f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:08.796755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739908a6-f8f2-49f7-9a7c-86f4dbbc803f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:08.802171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 547/1117] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:01:08 2022] GET /identity/v3/role_assignments?user.id=6b50481f1a1e4adbaed7404fddd75280&scope.project.id=877f67a84bb541eab9460478b8186707&role.id=1ff35caa6d284df39cd287349f66c567 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:10.793708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af08635d-f639-4d00-8030-37ca0ef17f43 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:10.794221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af08635d-f639-4d00-8030-37ca0ef17f43 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:10.794319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af08635d-f639-4d00-8030-37ca0ef17f43 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:10.795644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 571/1118] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:10.801545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6ad245-50cb-414b-9f99-d52ac3ec95b3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:10.801952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6ad245-50cb-414b-9f99-d52ac3ec95b3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:10.802083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec6ad245-50cb-414b-9f99-d52ac3ec95b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:10.819163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ec6ad245-50cb-414b-9f99-d52ac3ec95b3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:10.820327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ec6ad245-50cb-414b-9f99-d52ac3ec95b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:10.842933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:10.885206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 548/1119] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:10 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:10.908710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af08635d-f639-4d00-8030-37ca0ef17f43 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:10.930273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:11.006927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af08635d-f639-4d00-8030-37ca0ef17f43 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:11.039360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:11.040206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:11.040485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:11.040758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] PATH_INFO: `/v3/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:11.042209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] RBAC: Authorizing `identity:get_role(role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:11.043595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:11.044955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 572/1120] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:10 2022] GET /identity/v3/roles/1ff35caa6d284df39cd287349f66c567 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:11.051323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6ad245-50cb-414b-9f99-d52ac3ec95b3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:11.147060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec6ad245-50cb-414b-9f99-d52ac3ec95b3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:11.180098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:11.180952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:11.181229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:11.181501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] PATH_INFO: `/v3/users/6b50481f1a1e4adbaed7404fddd75280` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:11.182661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] RBAC: Authorizing `identity:get_user(user_id=6b50481f1a1e4adbaed7404fddd75280)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:11.184110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:11.185705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 549/1121] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:11 2022] GET /identity/v3/users/6b50481f1a1e4adbaed7404fddd75280 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:11.192256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:11.275944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:11.287662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-903326c1-2a22-408c-a150-321b815f6f6f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:11.321759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2380e7b-24e0-412d-9235-92283f66fe6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:11.322602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2380e7b-24e0-412d-9235-92283f66fe6a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:11.322909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2380e7b-24e0-412d-9235-92283f66fe6a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:11.323187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2380e7b-24e0-412d-9235-92283f66fe6a None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:11.324355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2380e7b-24e0-412d-9235-92283f66fe6a None admin] RBAC: Authorizing `identity:get_project(project_id=877f67a84bb541eab9460478b8186707)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:11.325786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2380e7b-24e0-412d-9235-92283f66fe6a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:11.327504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 573/1122] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:11 2022] GET /identity/v3/projects/877f67a84bb541eab9460478b8186707 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:11.333501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:11.361756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:11.432559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11566105-c900-43c7-b6fc-d4ce21a65c6d None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:11.464852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3b6af2-5afb-413b-b515-b99c6c717ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:11.465727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b6af2-5afb-413b-b515-b99c6c717ea9 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:11.466010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b6af2-5afb-413b-b515-b99c6c717ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:11.466289 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3b6af2-5afb-413b-b515-b99c6c717ea9 None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707/users/6b50481f1a1e4adbaed7404fddd75280/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:11.468467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b6af2-5afb-413b-b515-b99c6c717ea9 None admin] RBAC: Authorizing `identity:create_grant(project_id=877f67a84bb541eab9460478b8186707, user_id=6b50481f1a1e4adbaed7404fddd75280, role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:11.470558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3b6af2-5afb-413b-b515-b99c6c717ea9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:11.543108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 550/1123] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:01:11 2022] PUT /identity/v3/projects/877f67a84bb541eab9460478b8186707/users/6b50481f1a1e4adbaed7404fddd75280/roles/1ff35caa6d284df39cd287349f66c567 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:01:13.535874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640cc029-b0fd-4d96-a714-9c51b11c5a3b None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:13.536439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640cc029-b0fd-4d96-a714-9c51b11c5a3b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:13.536439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640cc029-b0fd-4d96-a714-9c51b11c5a3b None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:13.537804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 574/1124] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:13.542971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa1b67-0634-48ad-89a6-92b478fa027e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:13.543370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa1b67-0634-48ad-89a6-92b478fa027e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:13.543526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afa1b67-0634-48ad-89a6-92b478fa027e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:13.560347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-8afa1b67-0634-48ad-89a6-92b478fa027e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:13.561512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8afa1b67-0634-48ad-89a6-92b478fa027e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:13.626125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 551/1125] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:13 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:13.649299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640cc029-b0fd-4d96-a714-9c51b11c5a3b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:13.749790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640cc029-b0fd-4d96-a714-9c51b11c5a3b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:13.783174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:13.784025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:13.784315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:13.784588 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] PATH_INFO: `/v3/roles/1ff35caa6d284df39cd287349f66c567` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:13.786110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] RBAC: Authorizing `identity:get_role(role_id=1ff35caa6d284df39cd287349f66c567)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:13.787458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:13.788830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 575/1126] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:13 2022] GET /identity/v3/roles/1ff35caa6d284df39cd287349f66c567 => generated 223 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:13.794638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa1b67-0634-48ad-89a6-92b478fa027e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:13.812107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:13.890597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afa1b67-0634-48ad-89a6-92b478fa027e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:13.923900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:13.924758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:13.925036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:13.925307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] PATH_INFO: `/v3/users/6b50481f1a1e4adbaed7404fddd75280` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:13.926471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] RBAC: Authorizing `identity:get_user(user_id=6b50481f1a1e4adbaed7404fddd75280)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:13.927933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:13.929526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 552/1127] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:13 2022] GET /identity/v3/users/6b50481f1a1e4adbaed7404fddd75280 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:13.935225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:13.959954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:14.033364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64746e48-0030-44db-8ea5-0ca9069b8408 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:14.065202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:14.065777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:14.066063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:14.066340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:14.129807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:14.131275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:14.154520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:14.227763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 576/1128] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:13 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:14.233562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:14.315137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:14.331070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ab4d44-0f2c-4e5e-a12a-0e8044903ecb None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:14.363251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-640c8543-3f12-4adb-90f3-0df2658fe084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:14.364081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640c8543-3f12-4adb-90f3-0df2658fe084 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:14.364365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640c8543-3f12-4adb-90f3-0df2658fe084 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:14.364629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-640c8543-3f12-4adb-90f3-0df2658fe084 None admin] PATH_INFO: `/v3/projects/877f67a84bb541eab9460478b8186707` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:14.365788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640c8543-3f12-4adb-90f3-0df2658fe084 None admin] RBAC: Authorizing `identity:get_project(project_id=877f67a84bb541eab9460478b8186707)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:14.367242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-640c8543-3f12-4adb-90f3-0df2658fe084 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:14.368889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 553/1129] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:14 2022] GET /identity/v3/projects/877f67a84bb541eab9460478b8186707 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:14.374932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:14.471007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1754682-861e-45e5-8a4d-40b24b8aca91 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:14.492700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:14.503489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7856b8d4-8b3f-46f6-8167-fdfb129858f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:14.504125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856b8d4-8b3f-46f6-8167-fdfb129858f2 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:14.504401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856b8d4-8b3f-46f6-8167-fdfb129858f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:14.504676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7856b8d4-8b3f-46f6-8167-fdfb129858f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:14.505518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856b8d4-8b3f-46f6-8167-fdfb129858f2 None admin] RBAC: Adding query filter params (role.id=1ff35caa6d284df39cd287349f66c567, scope.project.id=877f67a84bb541eab9460478b8186707, user.id=6b50481f1a1e4adbaed7404fddd75280) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:14.505823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856b8d4-8b3f-46f6-8167-fdfb129858f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:14.507266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7856b8d4-8b3f-46f6-8167-fdfb129858f2 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:14.512809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 577/1130] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:01:14 2022] GET /identity/v3/role_assignments?user.id=6b50481f1a1e4adbaed7404fddd75280&scope.project.id=877f67a84bb541eab9460478b8186707&role.id=1ff35caa6d284df39cd287349f66c567 => generated 608 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:01:16.537681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91890ec7-714b-427f-98b3-71379ce87c7b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:16.538222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91890ec7-714b-427f-98b3-71379ce87c7b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:16.538222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91890ec7-714b-427f-98b3-71379ce87c7b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:16.539621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 554/1131] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:16.545064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3b3fe-eb46-4a4d-876a-0fccb2cd203e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:16.545487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3b3fe-eb46-4a4d-876a-0fccb2cd203e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:16.545618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad3b3fe-eb46-4a4d-876a-0fccb2cd203e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:16.563422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4ad3b3fe-eb46-4a4d-876a-0fccb2cd203e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:16.564761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4ad3b3fe-eb46-4a4d-876a-0fccb2cd203e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:16.629803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 578/1132] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:16 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:16.653653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91890ec7-714b-427f-98b3-71379ce87c7b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:16.758376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91890ec7-714b-427f-98b3-71379ce87c7b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:16.791588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0187a0dd-98ef-496d-9e63-92c9c46b71fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:16.792306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187a0dd-98ef-496d-9e63-92c9c46b71fd None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:16.792588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187a0dd-98ef-496d-9e63-92c9c46b71fd None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:16.792863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0187a0dd-98ef-496d-9e63-92c9c46b71fd None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:16.794020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0187a0dd-98ef-496d-9e63-92c9c46b71fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:16.795476 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0187a0dd-98ef-496d-9e63-92c9c46b71fd None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:16.797758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 555/1133] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:01:16 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:16.803902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad3b3fe-eb46-4a4d-876a-0fccb2cd203e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:16.898306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad3b3fe-eb46-4a4d-876a-0fccb2cd203e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:16.915454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:16.931911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97df9d63-8009-451f-9e43-3a4522133ccd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:16.932569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97df9d63-8009-451f-9e43-3a4522133ccd None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:16.932850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97df9d63-8009-451f-9e43-3a4522133ccd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:16.933124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97df9d63-8009-451f-9e43-3a4522133ccd None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:16.933833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97df9d63-8009-451f-9e43-3a4522133ccd None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:16.935298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97df9d63-8009-451f-9e43-3a4522133ccd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:17.007514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 579/1134] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:01:16 2022] POST /identity/v3/projects => generated 311 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:01:19.053513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4640a8ee-712d-4b35-81c4-f5ed51e5a35a None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:19.054043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4640a8ee-712d-4b35-81c4-f5ed51e5a35a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:19.054043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4640a8ee-712d-4b35-81c4-f5ed51e5a35a None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:19.055366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 556/1135] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:19.060377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16859faa-10a5-4347-a265-b528f4c629b1 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:19.060793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16859faa-10a5-4347-a265-b528f4c629b1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:19.060947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16859faa-10a5-4347-a265-b528f4c629b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:19.077990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-16859faa-10a5-4347-a265-b528f4c629b1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:19.079791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-16859faa-10a5-4347-a265-b528f4c629b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:19.145490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 580/1136] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:19 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:19.170101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4640a8ee-712d-4b35-81c4-f5ed51e5a35a None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:19.192761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:19.270573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4640a8ee-712d-4b35-81c4-f5ed51e5a35a None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:19.303066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b2ab5f-a30f-45e8-97a4-96bd54a929e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:19.303879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b2ab5f-a30f-45e8-97a4-96bd54a929e1 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:19.304111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b2ab5f-a30f-45e8-97a4-96bd54a929e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:19.304380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b2ab5f-a30f-45e8-97a4-96bd54a929e1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:19.305526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b2ab5f-a30f-45e8-97a4-96bd54a929e1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:19.306972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b2ab5f-a30f-45e8-97a4-96bd54a929e1 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:19.308592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 557/1137] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:01:19 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:19.314879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16859faa-10a5-4347-a265-b528f4c629b1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:19.406367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:19.409226 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16859faa-10a5-4347-a265-b528f4c629b1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:19.442959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2487603e-fee9-4f36-a93a-844253541c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:19.443609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2487603e-fee9-4f36-a93a-844253541c33 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:19.443887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2487603e-fee9-4f36-a93a-844253541c33 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:19.444158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2487603e-fee9-4f36-a93a-844253541c33 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:19.444863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2487603e-fee9-4f36-a93a-844253541c33 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:19.446278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2487603e-fee9-4f36-a93a-844253541c33 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:19.543793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 581/1138] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 12:01:19 2022] POST /identity/v3/users => generated 286 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:01:21.617462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19c0ade-4f52-4076-a398-0d404b4d6d2a None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:21.618031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19c0ade-4f52-4076-a398-0d404b4d6d2a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:21.618031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19c0ade-4f52-4076-a398-0d404b4d6d2a None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:21.619392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 558/1139] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:21.624394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3611c220-00fe-4cd9-aa07-4207856bb627 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:21.624803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3611c220-00fe-4cd9-aa07-4207856bb627 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:21.625625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3611c220-00fe-4cd9-aa07-4207856bb627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:21.642055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3611c220-00fe-4cd9-aa07-4207856bb627 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:21.643278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3611c220-00fe-4cd9-aa07-4207856bb627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:21.707551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 582/1140] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:21 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:21.731028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19c0ade-4f52-4076-a398-0d404b4d6d2a None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:21.817202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:21.829755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19c0ade-4f52-4076-a398-0d404b4d6d2a None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:21.863200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:21.863967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:21.864245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:21.864517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:21.874461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:01:21.882195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:21.883458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:01:21.884561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 559/1141] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:01:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:21.889320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3611c220-00fe-4cd9-aa07-4207856bb627 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:21.971876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:21.992881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3611c220-00fe-4cd9-aa07-4207856bb627 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:22.025808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:22.026548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:22.026939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:22.027234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:22.027994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:22.028312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:22.029722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:22.039120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 583/1142] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:01:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:22.044827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:22.141751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bf18be-0cfb-4379-8298-928412ad4acd None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:22.173960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:22.174830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:22.175106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:22.175375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] PATH_INFO: `/v3/users/b7905f8ae3a845a78ec6f39f4c332786` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:22.187918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] RBAC: Authorizing `identity:get_user(user_id=b7905f8ae3a845a78ec6f39f4c332786)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:22.189327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:22.190939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 560/1143] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:22 2022] GET /identity/v3/users/b7905f8ae3a845a78ec6f39f4c332786 => generated 286 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:22.196446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:22.294054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ceb896-812f-40e9-a637-3fae74db82ab None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:22.327095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:22.327734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:22.328542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:22.328977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:22.346559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:22.394189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:22.395759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:22.490286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 584/1144] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:22 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:22.496243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:22.504769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:22.594956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b222536-4cbe-493b-bace-e98fd3b01944 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:22.628171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f260216-8f28-465d-b363-e48445ab669a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:22.629052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f260216-8f28-465d-b363-e48445ab669a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:22.629334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f260216-8f28-465d-b363-e48445ab669a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:22.629608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f260216-8f28-465d-b363-e48445ab669a None admin] PATH_INFO: `/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:22.630796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f260216-8f28-465d-b363-e48445ab669a None admin] RBAC: Authorizing `identity:get_project(project_id=eb8c3ea62d7e45938b4875fae220b5f5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:22.632226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f260216-8f28-465d-b363-e48445ab669a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:22.633903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 561/1145] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:22 2022] GET /identity/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:22.639956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:22.689601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:22.740466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee94c595-b9e6-4d1e-860c-29fc6d234fe0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:22.774628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b59c667-61f1-4115-a5ae-d1a654080d74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:22.775304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59c667-61f1-4115-a5ae-d1a654080d74 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:22.775837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59c667-61f1-4115-a5ae-d1a654080d74 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:22.776111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59c667-61f1-4115-a5ae-d1a654080d74 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:22.776935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b59c667-61f1-4115-a5ae-d1a654080d74 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=eb8c3ea62d7e45938b4875fae220b5f5, user.id=b7905f8ae3a845a78ec6f39f4c332786) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:22.777247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b59c667-61f1-4115-a5ae-d1a654080d74 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:22.778673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b59c667-61f1-4115-a5ae-d1a654080d74 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:22.784079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 585/1146] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:01:22 2022] GET /identity/v3/role_assignments?user.id=b7905f8ae3a845a78ec6f39f4c332786&scope.project.id=eb8c3ea62d7e45938b4875fae220b5f5&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:24.799226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d5a12a-da4d-4ad0-a547-243f2fee3132 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:24.799758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d5a12a-da4d-4ad0-a547-243f2fee3132 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:24.799758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d5a12a-da4d-4ad0-a547-243f2fee3132 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:24.801123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 562/1147] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:24.806211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24931373-f1c0-43ac-a93d-7fca5b0de523 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:24.806650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24931373-f1c0-43ac-a93d-7fca5b0de523 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:24.806754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24931373-f1c0-43ac-a93d-7fca5b0de523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:24.823875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-24931373-f1c0-43ac-a93d-7fca5b0de523 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:24.825062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-24931373-f1c0-43ac-a93d-7fca5b0de523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:24.888844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 586/1148] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:24 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:24.911873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d5a12a-da4d-4ad0-a547-243f2fee3132 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:24.986701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:25.010995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d5a12a-da4d-4ad0-a547-243f2fee3132 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:25.044575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:25.045353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:25.045645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:25.045923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:25.056342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:01:25.063998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:25.065075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:01:25.066278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 563/1149] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:01:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:25.071364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24931373-f1c0-43ac-a93d-7fca5b0de523 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:25.113241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:25.167772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24931373-f1c0-43ac-a93d-7fca5b0de523 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:25.200653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:25.201309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:25.201593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:25.201871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:25.202607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:25.203448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:25.204913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:25.212706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 587/1150] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:01:25 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:25.218407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:25.314744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a8c4532-c15a-4b9d-9b81-9cf35520a1a0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:25.347573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:25.348427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:25.348710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:25.348984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] PATH_INFO: `/v3/users/b7905f8ae3a845a78ec6f39f4c332786` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:25.350154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] RBAC: Authorizing `identity:get_user(user_id=b7905f8ae3a845a78ec6f39f4c332786)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:25.351597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:25.353187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 564/1151] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:25 2022] GET /identity/v3/users/b7905f8ae3a845a78ec6f39f4c332786 => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:25.358942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:25.455616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa725694-af8e-4665-a0bc-03a357bf926c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:25.457859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:25.489087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f493b5b0-a8a4-4961-99bb-e5c1c6f7acec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:25.489953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493b5b0-a8a4-4961-99bb-e5c1c6f7acec None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:25.490237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493b5b0-a8a4-4961-99bb-e5c1c6f7acec None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:25.490513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f493b5b0-a8a4-4961-99bb-e5c1c6f7acec None admin] PATH_INFO: `/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:25.491708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f493b5b0-a8a4-4961-99bb-e5c1c6f7acec None admin] RBAC: Authorizing `identity:get_project(project_id=eb8c3ea62d7e45938b4875fae220b5f5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:25.493143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f493b5b0-a8a4-4961-99bb-e5c1c6f7acec None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:25.494961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 588/1152] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:25 2022] GET /identity/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:25.500970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:25.503863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:25.600602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-811615ff-a4d4-4392-8a3c-6f8fc46bf309 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:25.632984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6937adc0-2f73-4304-9f60-b0d5deb66427 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:25.633855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6937adc0-2f73-4304-9f60-b0d5deb66427 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:25.634136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6937adc0-2f73-4304-9f60-b0d5deb66427 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:25.634410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6937adc0-2f73-4304-9f60-b0d5deb66427 None admin] PATH_INFO: `/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5/users/b7905f8ae3a845a78ec6f39f4c332786/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:25.636376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6937adc0-2f73-4304-9f60-b0d5deb66427 None admin] RBAC: Authorizing `identity:create_grant(project_id=eb8c3ea62d7e45938b4875fae220b5f5, user_id=b7905f8ae3a845a78ec6f39f4c332786, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:25.637804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6937adc0-2f73-4304-9f60-b0d5deb66427 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:25.713893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 565/1153] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:01:25 2022] PUT /identity/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5/users/b7905f8ae3a845a78ec6f39f4c332786/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:01:27.691101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2c675-e556-48be-a36d-3d59302e7eb2 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:27.691682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2c675-e556-48be-a36d-3d59302e7eb2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:27.691682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cf2c675-e556-48be-a36d-3d59302e7eb2 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:27.693016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 589/1154] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:27.698096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c42d999-4b3d-4055-9ae4-e7d236f293f2 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:27.698535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c42d999-4b3d-4055-9ae4-e7d236f293f2 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:27.698660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c42d999-4b3d-4055-9ae4-e7d236f293f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:27.715571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7c42d999-4b3d-4055-9ae4-e7d236f293f2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:27.716762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7c42d999-4b3d-4055-9ae4-e7d236f293f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:27.772359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:27.783326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 566/1155] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:27 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:27.807153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf2c675-e556-48be-a36d-3d59302e7eb2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:27.907218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cf2c675-e556-48be-a36d-3d59302e7eb2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:27.940472 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:27.941301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:27.941535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:27.941811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:27.951738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:01:27.960183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:27.961117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:01:27.961999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 590/1156] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:01:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:27.967379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c42d999-4b3d-4055-9ae4-e7d236f293f2 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:28.061836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c42d999-4b3d-4055-9ae4-e7d236f293f2 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:28.095041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:28.095694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:28.095975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:28.096246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:28.096981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:28.097287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:28.098645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:28.106471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 567/1157] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:01:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:01:28.112485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:28.144243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:28.211777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33480b76-e1dd-4f46-b612-35791592e48b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:28.245059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:28.245902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:28.246185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:28.246461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] PATH_INFO: `/v3/users/b7905f8ae3a845a78ec6f39f4c332786` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:28.247662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] RBAC: Authorizing `identity:get_user(user_id=b7905f8ae3a845a78ec6f39f4c332786)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:28.249096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:28.251545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 591/1158] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:28 2022] GET /identity/v3/users/b7905f8ae3a845a78ec6f39f4c332786 => generated 286 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:28.257322 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:28.311149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:28.354547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d0ee5f-88a3-46a4-bcf3-737d0fc060b5 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:28.387527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:28.388161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:28.388442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:28.388717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:28.453454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:28.454917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:28.506686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:28.550327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 568/1159] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:28 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:28.556163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:28.650175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:28.653250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a8fd1bb-451d-4eb3-a8db-ef0062e58d94 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:28.687650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31637ff0-edaf-4dc1-b1bb-23a1bcd8b225 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:28.688569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31637ff0-edaf-4dc1-b1bb-23a1bcd8b225 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:28.688778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31637ff0-edaf-4dc1-b1bb-23a1bcd8b225 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:28.689052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31637ff0-edaf-4dc1-b1bb-23a1bcd8b225 None admin] PATH_INFO: `/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:28.690217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31637ff0-edaf-4dc1-b1bb-23a1bcd8b225 None admin] RBAC: Authorizing `identity:get_project(project_id=eb8c3ea62d7e45938b4875fae220b5f5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:28.691677 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31637ff0-edaf-4dc1-b1bb-23a1bcd8b225 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:28.693391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 592/1160] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:28 2022] GET /identity/v3/projects/eb8c3ea62d7e45938b4875fae220b5f5 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:28.699602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:28.795234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d447ae02-a443-4e08-854c-6bb632adb2a2 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:28.828789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d277a3-8c06-4f6b-8420-79f0e2cdb78b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:28.829588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d277a3-8c06-4f6b-8420-79f0e2cdb78b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:28.829870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d277a3-8c06-4f6b-8420-79f0e2cdb78b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:28.830148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d277a3-8c06-4f6b-8420-79f0e2cdb78b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:28.830997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d277a3-8c06-4f6b-8420-79f0e2cdb78b None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=eb8c3ea62d7e45938b4875fae220b5f5, user.id=b7905f8ae3a845a78ec6f39f4c332786) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:28.831311 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d277a3-8c06-4f6b-8420-79f0e2cdb78b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:28.832733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d277a3-8c06-4f6b-8420-79f0e2cdb78b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:28.838451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 569/1161] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:01:28 2022] GET /identity/v3/role_assignments?user.id=b7905f8ae3a845a78ec6f39f4c332786&scope.project.id=eb8c3ea62d7e45938b4875fae220b5f5&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:30.875605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779650a8-d318-4448-a01e-14bbdad2b802 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:30.876122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779650a8-d318-4448-a01e-14bbdad2b802 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:30.876211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779650a8-d318-4448-a01e-14bbdad2b802 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:30.877521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 593/1162] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:30.882645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774ee6ad-42ff-4c88-8fdc-4285b0e68b43 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:30.883065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774ee6ad-42ff-4c88-8fdc-4285b0e68b43 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:30.883240 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774ee6ad-42ff-4c88-8fdc-4285b0e68b43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:30.901175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-774ee6ad-42ff-4c88-8fdc-4285b0e68b43 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:30.902360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-774ee6ad-42ff-4c88-8fdc-4285b0e68b43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:30.915270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:30.968538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 570/1163] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:30 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:30.991131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779650a8-d318-4448-a01e-14bbdad2b802 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:31.090220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779650a8-d318-4448-a01e-14bbdad2b802 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:31.123827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:31.124465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:31.124777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:31.125051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:31.158060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:31.189957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:31.191425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:31.287032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 594/1164] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:30 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:31.292917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774ee6ad-42ff-4c88-8fdc-4285b0e68b43 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:31.386942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774ee6ad-42ff-4c88-8fdc-4285b0e68b43 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:31.419361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d1e36d-528e-46f4-b3ce-9926e39e0150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:31.420144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d1e36d-528e-46f4-b3ce-9926e39e0150 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:31.420422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d1e36d-528e-46f4-b3ce-9926e39e0150 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:31.420696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d1e36d-528e-46f4-b3ce-9926e39e0150 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:31.427699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d1e36d-528e-46f4-b3ce-9926e39e0150 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:31.429131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d1e36d-528e-46f4-b3ce-9926e39e0150 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-47d1e36d-528e-46f4-b3ce-9926e39e0150 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:31.443690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:31.444891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:01:31.446083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 571/1165] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:01:31 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:31.450722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:31.515855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:31.548195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa781ba2-a68e-4c37-81bc-5a5cd1d508a6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:31.580672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-517c2fab-53d7-4c8f-8a40-05f695261496 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:31.581332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c2fab-53d7-4c8f-8a40-05f695261496 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:31.581618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c2fab-53d7-4c8f-8a40-05f695261496 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:31.581898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-517c2fab-53d7-4c8f-8a40-05f695261496 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:31.582615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c2fab-53d7-4c8f-8a40-05f695261496 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:31.582957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c2fab-53d7-4c8f-8a40-05f695261496 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:31.584321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-517c2fab-53d7-4c8f-8a40-05f695261496 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:31.591332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 595/1166] 10.222.0.27 () {56 vars in 1115 bytes} [Thu Jul 14 12:01:31 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:33.604935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9275e-d834-4f71-a7ce-45a373e3cb07 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:33.605473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9275e-d834-4f71-a7ce-45a373e3cb07 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:33.605473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b9275e-d834-4f71-a7ce-45a373e3cb07 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:33.606920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 572/1167] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:33.612822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7ffe3-342b-495e-8c55-7ce5ef92d5da None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:33.613223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7ffe3-342b-495e-8c55-7ce5ef92d5da None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:33.613374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7ffe3-342b-495e-8c55-7ce5ef92d5da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:33.630418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-57a7ffe3-342b-495e-8c55-7ce5ef92d5da None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:33.631618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-57a7ffe3-342b-495e-8c55-7ce5ef92d5da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:33.696800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 596/1168] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:33 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:33.720010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b9275e-d834-4f71-a7ce-45a373e3cb07 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:33.731425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:33.820412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b9275e-d834-4f71-a7ce-45a373e3cb07 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:33.852643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971ba28e-0b2f-4a45-b850-14eeeee88c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:33.853287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ba28e-0b2f-4a45-b850-14eeeee88c79 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:33.853567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ba28e-0b2f-4a45-b850-14eeeee88c79 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:33.853840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ba28e-0b2f-4a45-b850-14eeeee88c79 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:33.854585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971ba28e-0b2f-4a45-b850-14eeeee88c79 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:33.856027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971ba28e-0b2f-4a45-b850-14eeeee88c79 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:33.922307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 573/1169] 10.222.0.27 () {60 vars in 1138 bytes} [Thu Jul 14 12:01:33 2022] POST /identity/v3/domains => generated 267 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:01:35.968030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107e08fe-ea8e-4b42-9302-ca770cce350e None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:35.968708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107e08fe-ea8e-4b42-9302-ca770cce350e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:35.968708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107e08fe-ea8e-4b42-9302-ca770cce350e None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:35.970163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 597/1170] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:35.975371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0c121-65e0-4cfb-b5aa-f431cdc196e3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:35.975775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0c121-65e0-4cfb-b5aa-f431cdc196e3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:35.975932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda0c121-65e0-4cfb-b5aa-f431cdc196e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:35.992786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-dda0c121-65e0-4cfb-b5aa-f431cdc196e3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:35.993899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dda0c121-65e0-4cfb-b5aa-f431cdc196e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:36.054133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:36.059364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 574/1171] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:35 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:36.083695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107e08fe-ea8e-4b42-9302-ca770cce350e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:36.145302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:36.184218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107e08fe-ea8e-4b42-9302-ca770cce350e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:36.216159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5920c83-18ee-434d-baf9-4914dda6ab9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:36.216998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5920c83-18ee-434d-baf9-4914dda6ab9a None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:36.217279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5920c83-18ee-434d-baf9-4914dda6ab9a None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:36.217555 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5920c83-18ee-434d-baf9-4914dda6ab9a None admin] PATH_INFO: `/v3/domains/10b326b6d9e846098358978dde8f7ba4` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:36.228457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5920c83-18ee-434d-baf9-4914dda6ab9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=10b326b6d9e846098358978dde8f7ba4)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:36.229890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5920c83-18ee-434d-baf9-4914dda6ab9a None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:36.231671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 598/1172] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Jul 14 12:01:36 2022] GET /identity/v3/domains/10b326b6d9e846098358978dde8f7ba4 => generated 267 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:36.238142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda0c121-65e0-4cfb-b5aa-f431cdc196e3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:36.332494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda0c121-65e0-4cfb-b5aa-f431cdc196e3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:36.365488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ae9210-f807-4fd2-98ac-7438aeeb57bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:36.366131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ae9210-f807-4fd2-98ac-7438aeeb57bd None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:36.366413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ae9210-f807-4fd2-98ac-7438aeeb57bd None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:36.366688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ae9210-f807-4fd2-98ac-7438aeeb57bd None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:36.367419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ae9210-f807-4fd2-98ac-7438aeeb57bd None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:36.368845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28ae9210-f807-4fd2-98ac-7438aeeb57bd None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:36.463404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 575/1173] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:01:36 2022] POST /identity/v3/projects => generated 361 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:01:38.515983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1613bce5-5902-4fcd-b7d6-352629600f61 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:38.516598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1613bce5-5902-4fcd-b7d6-352629600f61 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:38.516598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1613bce5-5902-4fcd-b7d6-352629600f61 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:38.517900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 599/1174] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:38.523068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538f4cda-f5b7-4ace-a010-3a7e7ff752bf None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:38.523466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538f4cda-f5b7-4ace-a010-3a7e7ff752bf None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:38.523640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-538f4cda-f5b7-4ace-a010-3a7e7ff752bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:38.540741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-538f4cda-f5b7-4ace-a010-3a7e7ff752bf None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:38.541927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-538f4cda-f5b7-4ace-a010-3a7e7ff752bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:38.592446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:38.613823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 576/1175] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:38 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:38.638458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1613bce5-5902-4fcd-b7d6-352629600f61 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:38.740341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1613bce5-5902-4fcd-b7d6-352629600f61 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:38.761788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:38.773481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefcee39-729e-4ac6-bc19-871da7179e91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:38.774275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefcee39-729e-4ac6-bc19-871da7179e91 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:38.774557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefcee39-729e-4ac6-bc19-871da7179e91 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:38.774860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefcee39-729e-4ac6-bc19-871da7179e91 None admin] PATH_INFO: `/v3/domains/10b326b6d9e846098358978dde8f7ba4` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:38.776283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefcee39-729e-4ac6-bc19-871da7179e91 None admin] RBAC: Authorizing `identity:get_domain(domain_id=10b326b6d9e846098358978dde8f7ba4)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:38.777717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefcee39-729e-4ac6-bc19-871da7179e91 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:38.779355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 600/1176] 10.222.0.27 () {56 vars in 1183 bytes} [Thu Jul 14 12:01:38 2022] GET /identity/v3/domains/10b326b6d9e846098358978dde8f7ba4 => generated 267 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:38.785686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538f4cda-f5b7-4ace-a010-3a7e7ff752bf None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:38.882069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-538f4cda-f5b7-4ace-a010-3a7e7ff752bf None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:38.914621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3dae1d-54a4-4e03-89fd-c8857294f088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:38.915309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3dae1d-54a4-4e03-89fd-c8857294f088 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:38.915607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3dae1d-54a4-4e03-89fd-c8857294f088 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:38.915879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3dae1d-54a4-4e03-89fd-c8857294f088 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:38.916592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3dae1d-54a4-4e03-89fd-c8857294f088 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:38.918014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad3dae1d-54a4-4e03-89fd-c8857294f088 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:39.023149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 577/1177] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 12:01:38 2022] POST /identity/v3/users => generated 311 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:01:41.076151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3634032-b9ca-4480-890f-60d8e124993c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:41.076670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3634032-b9ca-4480-890f-60d8e124993c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:41.076670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3634032-b9ca-4480-890f-60d8e124993c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:41.078872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 601/1178] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:41 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:41.083909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153d8071-4327-407a-80dd-f52a3904c3c3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:41.084343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153d8071-4327-407a-80dd-f52a3904c3c3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:41.084471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153d8071-4327-407a-80dd-f52a3904c3c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:41.101435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-153d8071-4327-407a-80dd-f52a3904c3c3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:41.102648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-153d8071-4327-407a-80dd-f52a3904c3c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:41.130295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:41.167760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 578/1179] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:41 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:41.191909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3634032-b9ca-4480-890f-60d8e124993c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:41.291831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3634032-b9ca-4480-890f-60d8e124993c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:41.324416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:41.325197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:41.325477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:41.325751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:41.336261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:01:41.343805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:41.344983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:01:41.346084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 602/1180] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:01:41 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:41.350927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153d8071-4327-407a-80dd-f52a3904c3c3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:41.445462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153d8071-4327-407a-80dd-f52a3904c3c3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:41.478018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:41.478669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:41.478976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:41.479253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:41.480031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:41.480339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:41.481700 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:41.489518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 579/1181] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:01:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:41.495138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:41.544477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:41.593064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95dbbe94-ba9d-4286-a6f5-61aa14669e5e None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:41.669579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:41.670427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:41.670704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:41.671003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] PATH_INFO: `/v3/users/3861b1db97474a36a7081c699ec478e1` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:41.684185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] RBAC: Authorizing `identity:get_user(user_id=3861b1db97474a36a7081c699ec478e1)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:41.685608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:41.687920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 603/1182] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:41 2022] GET /identity/v3/users/3861b1db97474a36a7081c699ec478e1 => generated 311 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:41.693613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:41.721296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:41.791107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb7e5d6-99d2-42eb-abb6-e178057bcc61 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:41.823967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:41.824604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:41.824887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:41.825162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:41.889779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:41.891240 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:41.916902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:41.986496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 580/1183] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:41 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:41.992365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:42.089410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3641036f-1228-491a-b2f9-5c4f17344c26 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:42.099588 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:42.124012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a57f8af1-1c79-4d45-a57e-f8fe2ac9b144 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:42.124904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f8af1-1c79-4d45-a57e-f8fe2ac9b144 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:42.125213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f8af1-1c79-4d45-a57e-f8fe2ac9b144 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:42.125481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a57f8af1-1c79-4d45-a57e-f8fe2ac9b144 None admin] PATH_INFO: `/v3/projects/9ebc541abf6e451bbda5448923c413fc` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:42.126651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57f8af1-1c79-4d45-a57e-f8fe2ac9b144 None admin] RBAC: Authorizing `identity:get_project(project_id=9ebc541abf6e451bbda5448923c413fc)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:42.128119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a57f8af1-1c79-4d45-a57e-f8fe2ac9b144 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:42.129801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 604/1184] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:41 2022] GET /identity/v3/projects/9ebc541abf6e451bbda5448923c413fc => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:42.135807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:42.231711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1853d38d-28e0-4f81-bfa1-949c9c30244c None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:42.257689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:42.265754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22cd769d-58df-483d-80b2-e93c55be0974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:42.276421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cd769d-58df-483d-80b2-e93c55be0974 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:42.276421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cd769d-58df-483d-80b2-e93c55be0974 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:42.276421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22cd769d-58df-483d-80b2-e93c55be0974 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:42.276421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cd769d-58df-483d-80b2-e93c55be0974 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=9ebc541abf6e451bbda5448923c413fc, user.id=3861b1db97474a36a7081c699ec478e1) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:42.276421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cd769d-58df-483d-80b2-e93c55be0974 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:42.276421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22cd769d-58df-483d-80b2-e93c55be0974 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:42.276890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 581/1185] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:01:42 2022] GET /identity/v3/role_assignments?user.id=3861b1db97474a36a7081c699ec478e1&scope.project.id=9ebc541abf6e451bbda5448923c413fc&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:44.268104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe799c-2ff1-493f-9d13-4c191d23c5f9 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:44.268666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe799c-2ff1-493f-9d13-4c191d23c5f9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:44.268666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efe799c-2ff1-493f-9d13-4c191d23c5f9 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:44.270024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 605/1186] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:44.275188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2a5a2c-8583-44d0-9913-c17e4a6daac1 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:44.275574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2a5a2c-8583-44d0-9913-c17e4a6daac1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:44.275752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2a5a2c-8583-44d0-9913-c17e4a6daac1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:44.292976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-fd2a5a2c-8583-44d0-9913-c17e4a6daac1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:44.294163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fd2a5a2c-8583-44d0-9913-c17e4a6daac1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:44.357922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 582/1187] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:44 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:44.380660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efe799c-2ff1-493f-9d13-4c191d23c5f9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:44.478301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efe799c-2ff1-493f-9d13-4c191d23c5f9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:44.510592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:44.511388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:44.511670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:44.511940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:44.522231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:01:44.529899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:44.531027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:01:44.532160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 606/1188] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:01:44 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:44.537366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2a5a2c-8583-44d0-9913-c17e4a6daac1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:44.633398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2a5a2c-8583-44d0-9913-c17e4a6daac1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:44.656344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:44.667186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:44.673001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:44.673001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:44.673001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:44.673001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:44.673001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:44.673001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:44.679529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 583/1189] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:01:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:44.685268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:44.720428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:44.784592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b8e875-771a-4b57-91c7-5e1e3e864272 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:44.818174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:44.819059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:44.819335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:44.819603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] PATH_INFO: `/v3/users/3861b1db97474a36a7081c699ec478e1` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:44.820765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] RBAC: Authorizing `identity:get_user(user_id=3861b1db97474a36a7081c699ec478e1)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:44.822193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:44.823833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 607/1190] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:44 2022] GET /identity/v3/users/3861b1db97474a36a7081c699ec478e1 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:44.829830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:44.926409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb8a2094-a8ae-4899-9270-e91d1b8d9051 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:44.958816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712429dc-479e-4da2-9261-e053db1e1a28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:44.959655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712429dc-479e-4da2-9261-e053db1e1a28 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:44.959936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712429dc-479e-4da2-9261-e053db1e1a28 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:44.960205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712429dc-479e-4da2-9261-e053db1e1a28 None admin] PATH_INFO: `/v3/projects/9ebc541abf6e451bbda5448923c413fc` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:44.961356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712429dc-479e-4da2-9261-e053db1e1a28 None admin] RBAC: Authorizing `identity:get_project(project_id=9ebc541abf6e451bbda5448923c413fc)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:44.962795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712429dc-479e-4da2-9261-e053db1e1a28 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:44.964451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 584/1191] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:44 2022] GET /identity/v3/projects/9ebc541abf6e451bbda5448923c413fc => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:44.970550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:45.068169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e92ffbd2-45f0-4c40-acb3-ea50b3041a43 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:45.070350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:45.101015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76805dd-785b-40e3-8734-344ae141d150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:45.101895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76805dd-785b-40e3-8734-344ae141d150 None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:45.102194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76805dd-785b-40e3-8734-344ae141d150 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:45.102510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76805dd-785b-40e3-8734-344ae141d150 None admin] PATH_INFO: `/v3/projects/9ebc541abf6e451bbda5448923c413fc/users/3861b1db97474a36a7081c699ec478e1/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:45.104980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76805dd-785b-40e3-8734-344ae141d150 None admin] RBAC: Authorizing `identity:create_grant(project_id=9ebc541abf6e451bbda5448923c413fc, user_id=3861b1db97474a36a7081c699ec478e1, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:45.106411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76805dd-785b-40e3-8734-344ae141d150 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:45.171751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 608/1192] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:01:44 2022] PUT /identity/v3/projects/9ebc541abf6e451bbda5448923c413fc/users/3861b1db97474a36a7081c699ec478e1/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:01:47.161084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0ad59-2361-4169-8fe7-9eb6fd4823a5 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:47.161606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0ad59-2361-4169-8fe7-9eb6fd4823a5 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:47.161606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f0ad59-2361-4169-8fe7-9eb6fd4823a5 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:47.162962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 585/1193] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:01:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:01:47.167957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73d7574-4d5f-454c-8304-edfaa79a2b02 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:47.168385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73d7574-4d5f-454c-8304-edfaa79a2b02 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:47.168516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b73d7574-4d5f-454c-8304-edfaa79a2b02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:47.184971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b73d7574-4d5f-454c-8304-edfaa79a2b02 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:01:47.186152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b73d7574-4d5f-454c-8304-edfaa79a2b02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:01:47.250475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 609/1194] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:01:47 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:01:47.274176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0ad59-2361-4169-8fe7-9eb6fd4823a5 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:47.328223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:47.376676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f0ad59-2361-4169-8fe7-9eb6fd4823a5 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:47.409932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:47.410710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:47.411014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:47.411334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:47.420925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:01:47.428509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:01:47.429606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:01:47.430464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 586/1195] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:01:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:01:47.435929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73d7574-4d5f-454c-8304-edfaa79a2b02 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:47.525952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:47.532562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b73d7574-4d5f-454c-8304-edfaa79a2b02 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:47.566072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:47.566735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:47.567050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:47.567334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:47.568072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:47.568380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:47.569743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:47.578171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 610/1196] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:01:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:47.584062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:47.681635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d09397-cdb8-4373-9268-b2f98e2a4b03 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:47.694743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:47.715214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:47.716098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:47.716608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:47.717160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] PATH_INFO: `/v3/users/3861b1db97474a36a7081c699ec478e1` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:47.718521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] RBAC: Authorizing `identity:get_user(user_id=3861b1db97474a36a7081c699ec478e1)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:47.720078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:47.721773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 587/1197] 10.222.0.27 () {56 vars in 1177 bytes} [Thu Jul 14 12:01:47 2022] GET /identity/v3/users/3861b1db97474a36a7081c699ec478e1 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:47.727448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:47.825401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36dd5f8-b0dc-4669-8baa-e5a550e17be6 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:47.857084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:47.857720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:47.858001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:47.858279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:47.882644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:47.924692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:47.926128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:48.019973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 611/1198] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:01:47 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:01:48.025996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:48.127595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd552ea-9558-46f7-9b73-e5ee547590a0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:48.159653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f32b65b-c015-40cb-833c-bdcb3fec8a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:48.160492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f32b65b-c015-40cb-833c-bdcb3fec8a26 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:48.160794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f32b65b-c015-40cb-833c-bdcb3fec8a26 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:48.161063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f32b65b-c015-40cb-833c-bdcb3fec8a26 None admin] PATH_INFO: `/v3/projects/9ebc541abf6e451bbda5448923c413fc` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:48.162207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f32b65b-c015-40cb-833c-bdcb3fec8a26 None admin] RBAC: Authorizing `identity:get_project(project_id=9ebc541abf6e451bbda5448923c413fc)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:48.163656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f32b65b-c015-40cb-833c-bdcb3fec8a26 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:48.165305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 588/1199] 10.222.0.27 () {56 vars in 1186 bytes} [Thu Jul 14 12:01:48 2022] GET /identity/v3/projects/9ebc541abf6e451bbda5448923c413fc => generated 361 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:48.172159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:01:48.225712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:48.269076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0dad28-f8be-4016-b956-65d4e32c5e1c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:01:48.302666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab1d217-0b4a-408d-94d1-15ae56202fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:01:48.303280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab1d217-0b4a-408d-94d1-15ae56202fd5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:01:48.303560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab1d217-0b4a-408d-94d1-15ae56202fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:01:48.303833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab1d217-0b4a-408d-94d1-15ae56202fd5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:01:48.304664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab1d217-0b4a-408d-94d1-15ae56202fd5 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=9ebc541abf6e451bbda5448923c413fc, user.id=3861b1db97474a36a7081c699ec478e1) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:01:48.304976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab1d217-0b4a-408d-94d1-15ae56202fd5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:01:48.306410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab1d217-0b4a-408d-94d1-15ae56202fd5 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:01:48.312317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 612/1200] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:01:48 2022] GET /identity/v3/role_assignments?user.id=3861b1db97474a36a7081c699ec478e1&scope.project.id=9ebc541abf6e451bbda5448923c413fc&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:01:58.155747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=76 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:01:58.309552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=40 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:00.012268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d7490-668b-498c-9663-e980d331c2fb None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:00.012816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d7490-668b-498c-9663-e980d331c2fb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:00.012816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e8d7490-668b-498c-9663-e980d331c2fb None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:00.014115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 589/1201] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:00.020199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cbb85e-abd8-46c3-8b7a-b4a3f6a84ddc None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:00.020687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cbb85e-abd8-46c3-8b7a-b4a3f6a84ddc None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:00.020789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07cbb85e-abd8-46c3-8b7a-b4a3f6a84ddc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:00.042489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-07cbb85e-abd8-46c3-8b7a-b4a3f6a84ddc None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:00.043714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-07cbb85e-abd8-46c3-8b7a-b4a3f6a84ddc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:00.108373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 613/1202] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:00 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:00.133874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d7490-668b-498c-9663-e980d331c2fb None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:00.236113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e8d7490-668b-498c-9663-e980d331c2fb None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:00.269527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:00.270232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:00.270516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:00.270823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:00.281782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:00.283267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:00.301431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:00.302413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:00.303793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:00.303793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:00.303793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:00.303793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:00.303793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 590/1203] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:00.308646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cbb85e-abd8-46c3-8b7a-b4a3f6a84ddc None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:00.411199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07cbb85e-abd8-46c3-8b7a-b4a3f6a84ddc None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:00.439015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:00.444873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4c48fe-9828-42bb-81b8-492e2becc092 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:00.445531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4c48fe-9828-42bb-81b8-492e2becc092 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:00.445814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4c48fe-9828-42bb-81b8-492e2becc092 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:00.446091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4c48fe-9828-42bb-81b8-492e2becc092 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:00.446830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4c48fe-9828-42bb-81b8-492e2becc092 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:00.451187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4c48fe-9828-42bb-81b8-492e2becc092 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:00.451187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4c48fe-9828-42bb-81b8-492e2becc092 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:00.459917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 614/1204] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:00.465867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:00.519118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:00.565998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e85e162a-4736-4914-b6ff-e5fef48f1fef None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:00.600133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4924a9be-be43-4311-a7d9-08f01cb415f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:00.600795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4924a9be-be43-4311-a7d9-08f01cb415f7 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:00.601077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4924a9be-be43-4311-a7d9-08f01cb415f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:00.601349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4924a9be-be43-4311-a7d9-08f01cb415f7 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:00.602045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4924a9be-be43-4311-a7d9-08f01cb415f7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:00.603504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4924a9be-be43-4311-a7d9-08f01cb415f7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:00.685814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 591/1205] 10.222.0.27 () {60 vars in 1132 bytes} [Thu Jul 14 12:03:00 2022] POST /identity/v3/users => generated 251 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:03:02.735804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23001062-a03d-4126-9047-0827269d3e86 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:02.736236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23001062-a03d-4126-9047-0827269d3e86 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:02.736330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23001062-a03d-4126-9047-0827269d3e86 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:02.737754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 615/1206] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:02.742844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2fe07-bae1-4b96-87eb-0bba46209101 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:02.746776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2fe07-bae1-4b96-87eb-0bba46209101 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:02.746776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c2fe07-bae1-4b96-87eb-0bba46209101 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:02.760180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-65c2fe07-bae1-4b96-87eb-0bba46209101 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:02.761379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-65c2fe07-bae1-4b96-87eb-0bba46209101 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:02.828603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 592/1207] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:02 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:02.853457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23001062-a03d-4126-9047-0827269d3e86 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:02.956861 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23001062-a03d-4126-9047-0827269d3e86 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:02.988659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:02.989410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:02.989686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:02.989956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:03.000172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:03:03.008200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:03.009362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:03.010439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 616/1208] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:03:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:03.015579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c2fe07-bae1-4b96-87eb-0bba46209101 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:03.054934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:03.110872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c2fe07-bae1-4b96-87eb-0bba46209101 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:03.143083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:03.144457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:03.144739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:03.145010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:03.145745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:03.146045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:03.147425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:03.155813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 593/1209] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:03:03 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:03.161622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:03.212546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:03.261645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8b387b9-985b-4b6c-ac9b-d75ccbcddd24 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:03.295439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:03.296221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:03.296529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:03.296780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:03.307017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:03.308501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:03.326399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:03.327407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:03.328309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:03.328309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:03.328309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:03.328309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:03.328309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 617/1210] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:03.333710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:03.430781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:03.433690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c58d4fa-7670-48ed-ae64-32cf8306baaf None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:03.466955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:03.467613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:03.467897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:03.468176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:03.469017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:03.469331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:03.470691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:03.483323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 594/1211] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:03.489753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:03.590162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dccb8fd2-2fe2-498c-8c14-f11e599dc808 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:03.591627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:03.623728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:03.624518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:03.624801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:03.625078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:03.633456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:03.634938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:03:03.649983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:03.651125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:03.652454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 618/1212] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:03 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:03.657420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:03.755557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b5c37ad-56a2-437d-b82a-b5c72af1ae68 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:03.788397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:03.789051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:03.789329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:03.789598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:03.790423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:03.790727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:03.792160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:03.808527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 595/1213] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Jul 14 12:03:03 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:03.814215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:03.915446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dc234f-42fb-4793-85b4-2e4c2b052d77 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:03.949228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:03.949889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:03.950168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:03.950449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:03.980132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:04.017515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:04.019869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:04.116972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 619/1214] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:03:03 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 303 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:03:04.122749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:04.141264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:04.219860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad11e1b-0d05-42af-a5d9-1fe0bc8c039f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:04.254900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:04.255684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:04.255962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:04.256232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:04.265983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:04.267454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:04.285951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:04.287058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:04.288228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:04.288228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:04.288228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:04.288228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:04.288228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 596/1215] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:04.292849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:04.354672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:04.391335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b83ea1ec-332b-411b-9fad-c3804bbff21c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:04.424631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:04.425337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:04.425614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:04.425885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:04.426595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:04.426925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:04.428407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:04.439965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 620/1216] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:04.446269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:04.519156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:04.548707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1736dd93-1c25-409b-a39f-caa6c8e2db66 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:04.583246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:04.584034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:04.584317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:04.584596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:04.591295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:04.595544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:03:04.606804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:04.607891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:04.614218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 597/1217] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:03:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:04.615056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:04.714517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07248a72-6961-428d-bd24-7bb4e85fdda9 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:04.737395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:04.748820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af220ec-6709-4784-80ca-37b360050d97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:04.749480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af220ec-6709-4784-80ca-37b360050d97 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:04.749763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af220ec-6709-4784-80ca-37b360050d97 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:04.750039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af220ec-6709-4784-80ca-37b360050d97 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:04.750851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af220ec-6709-4784-80ca-37b360050d97 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:04.751164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af220ec-6709-4784-80ca-37b360050d97 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:04.753094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af220ec-6709-4784-80ca-37b360050d97 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:04.764451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 621/1218] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:03:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:03:04.771563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:04.871202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93bb612-540c-4ee7-b239-66f183238da9 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:04.899367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:04.905447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168be1e0-66a3-40e0-8147-250ef62949e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:04.906096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168be1e0-66a3-40e0-8147-250ef62949e9 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:04.906382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168be1e0-66a3-40e0-8147-250ef62949e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:04.906663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168be1e0-66a3-40e0-8147-250ef62949e9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:04.907582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168be1e0-66a3-40e0-8147-250ef62949e9 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=a9c50ae1ba494c20b6133a96dfba21b3) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:04.907945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168be1e0-66a3-40e0-8147-250ef62949e9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:04.909806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168be1e0-66a3-40e0-8147-250ef62949e9 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:04.915360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 598/1219] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:03:04 2022] GET /identity/v3/role_assignments?user.id=a9c50ae1ba494c20b6133a96dfba21b3&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:06.931854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912147b-3673-4ed5-b520-120a5978cb34 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:06.932407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912147b-3673-4ed5-b520-120a5978cb34 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:06.932407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3912147b-3673-4ed5-b520-120a5978cb34 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:06.933788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 622/1220] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:06.939060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792099-b22e-44d6-ac5e-1cc7b5d3f991 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:06.939479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792099-b22e-44d6-ac5e-1cc7b5d3f991 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:06.939615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86792099-b22e-44d6-ac5e-1cc7b5d3f991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:06.957620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-86792099-b22e-44d6-ac5e-1cc7b5d3f991 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:06.958962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-86792099-b22e-44d6-ac5e-1cc7b5d3f991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:07.024885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 599/1221] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:07.047738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3912147b-3673-4ed5-b520-120a5978cb34 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:07.148308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3912147b-3673-4ed5-b520-120a5978cb34 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:07.182494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:07.183317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:07.183600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:07.183879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:07.193958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:03:07.201673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:07.202844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:07.204074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 623/1222] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:03:07 2022] GET /identity/v3/roles/service => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:07.209632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792099-b22e-44d6-ac5e-1cc7b5d3f991 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:07.310565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86792099-b22e-44d6-ac5e-1cc7b5d3f991 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:07.334894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:07.345440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:07.346104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:07.347033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:07.347314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:07.348056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] RBAC: Adding query filter params (name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:07.348366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:07.349734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:07.357608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 600/1223] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:03:07 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:07.363417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:07.406380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:07.462531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe43cae4-c55c-4208-b044-abf79759a264 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:07.494475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:07.495742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:07.495742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:07.495911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:07.506515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:07.507980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:07.526266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:07.530621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:07.531770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:07.531770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:07.531770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:07.531770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:07.531770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 624/1224] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:07.533753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:07.635514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8fd023-6e38-4724-8ad4-3e9d05d714cb None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:07.669876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:07.670533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:07.670849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:07.671129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:07.671855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:07.672165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:07.673529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:07.685762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 601/1225] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:07.692237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:07.791616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:07.799026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1eac6d-cde3-4f9d-b4b2-632177dc452d None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:07.842357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:07.843312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:07.843753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:07.844299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:07.852777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:07.854310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:03:07.870487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:07.871701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:07.872735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 625/1226] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:07.878303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:07.929377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:07.982266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9eeb546-3352-4be7-93d6-bf1ecb532656 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:08.017812 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:08.018558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:08.018864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:08.019143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:08.019977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:08.020288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:08.021801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:08.040375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 602/1227] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Jul 14 12:03:07 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:08.046235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:08.148606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9bdc91-c6dc-4677-b13a-dc057ef0c895 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:08.186248 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:08.186979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:08.187258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:08.187530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:08.199507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:08.202248 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:08.220474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:08.222069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:08.223126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:08.223126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:08.223126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:08.223126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:08.223126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 626/1228] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:03:08.228666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:08.332018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa957a4-dffe-4796-939b-5bfa77840bd4 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:08.334364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:08.370824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:08.371480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:08.371760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:08.372034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:08.372749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:08.373053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:08.374506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:08.387049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 603/1229] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:08.393482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:08.402695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:08.491377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac36afdc-8edd-4b7d-978e-0d3129faad21 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:08.524068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:08.524843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:08.525119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:08.525389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:08.532247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:08.533669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:03:08.548505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:08.550093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:08.551030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 627/1230] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:03:08 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:08.556681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:08.657318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03477e8e-af14-4a24-873f-3a30af69e5a0 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:08.691086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9a8aa30-4652-44e1-aa45-746052640714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:08.697559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8aa30-4652-44e1-aa45-746052640714 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:08.697559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8aa30-4652-44e1-aa45-746052640714 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:08.697559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9a8aa30-4652-44e1-aa45-746052640714 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:08.697559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8aa30-4652-44e1-aa45-746052640714 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:08.697559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8aa30-4652-44e1-aa45-746052640714 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:08.697559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9a8aa30-4652-44e1-aa45-746052640714 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:08.707475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 604/1231] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:03:08 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:08.713611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:08.779699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:08.814256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1412f6a-d809-488a-8d1d-2d0d0f18cea1 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:08.847579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90fb0b8-d28c-417e-a9f0-18223a243e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:08.848458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90fb0b8-d28c-417e-a9f0-18223a243e29 None admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:08.848744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90fb0b8-d28c-417e-a9f0-18223a243e29 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:08.849019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90fb0b8-d28c-417e-a9f0-18223a243e29 None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/a9c50ae1ba494c20b6133a96dfba21b3/roles/0fd45f27e71d455da928a71546fbe85e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:08.862977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90fb0b8-d28c-417e-a9f0-18223a243e29 None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=a9c50ae1ba494c20b6133a96dfba21b3, role_id=0fd45f27e71d455da928a71546fbe85e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:08.864485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e90fb0b8-d28c-417e-a9f0-18223a243e29 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:08.949675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 628/1232] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:03:08 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/a9c50ae1ba494c20b6133a96dfba21b3/roles/0fd45f27e71d455da928a71546fbe85e => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:03:10.942085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120f36b-2df2-491e-87f4-93bcac32c260 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:10.942691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120f36b-2df2-491e-87f4-93bcac32c260 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:10.942691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9120f36b-2df2-491e-87f4-93bcac32c260 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:10.944100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 605/1233] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:10.949115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21292e7b-2a34-4620-91d5-2a3853f6d91a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:10.949540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21292e7b-2a34-4620-91d5-2a3853f6d91a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:10.949664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21292e7b-2a34-4620-91d5-2a3853f6d91a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:10.967355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-21292e7b-2a34-4620-91d5-2a3853f6d91a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:10.968527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-21292e7b-2a34-4620-91d5-2a3853f6d91a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:11.033707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 629/1234] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:10 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:11.057198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9120f36b-2df2-491e-87f4-93bcac32c260 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:11.082642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:11.157600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9120f36b-2df2-491e-87f4-93bcac32c260 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:11.191761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:11.192632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:11.192914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:11.193183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] PATH_INFO: `/v3/roles/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:11.203091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:03:11.210905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:11.216100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:11.217385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 606/1235] 10.222.0.27 () {56 vars in 1102 bytes} [Thu Jul 14 12:03:11 2022] GET /identity/v3/roles/service => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:11.218339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21292e7b-2a34-4620-91d5-2a3853f6d91a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:11.264043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:11.315459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21292e7b-2a34-4620-91d5-2a3853f6d91a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:11.349424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:11.350108 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:11.350389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:11.350663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:11.351449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] RBAC: Adding query filter params (name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:11.351759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:11.353128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:11.360854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 630/1236] 10.222.0.27 () {56 vars in 1103 bytes} [Thu Jul 14 12:03:11 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:11.367093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:11.450347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:11.465450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b86d2a0-f053-4d48-98a7-7843002881ee None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:11.499995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:11.500786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:11.501065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:11.501335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:11.511597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:11.513150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:11.530125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:11.539685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:11.540719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:11.540719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:11.540719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:11.540719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:11.540719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 607/1237] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:11.541015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:11.638818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c517feaa-0ccc-48ca-afb7-496703d02398 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:11.647995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:11.672497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:11.673157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:11.673440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:11.673716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:11.674439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:11.674750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:11.682171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:11.687526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 631/1238] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:11.693361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:11.793388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ca4987-156a-47f4-be2b-10133f72593a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:11.825871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:11.826669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:11.827006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:11.827277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] PATH_INFO: `/v3/users/placement` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:11.830001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:11.835745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:11.838567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:03:11.851675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:11.852765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:11.853948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 608/1239] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:11 2022] GET /identity/v3/users/placement => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:11.858693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:11.958819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ab9e7e-b50e-435a-bb41-bc732c96f217 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:11.993303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:11.993967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:11.994333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:11.994611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:11.995498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:11.995811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:11.997274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:12.013984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 632/1240] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Jul 14 12:03:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:12.020025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:12.118225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7707802-8485-4f22-a0d0-8612bccbe86a None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:12.150986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:12.151666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:12.151955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:12.152236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:12.213633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:12.220006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:12.221982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:12.319921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 609/1241] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:03:12 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:03:12.325645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:12.353478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:12.423860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5a23b0-4138-40c8-82d3-3655d7e73736 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:12.456919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:12.457760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:12.458043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:12.458316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:12.468184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:12.469569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:12.489239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:12.490392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:12.491657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:12.491657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:12.491657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:12.491657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:12.491657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 633/1242] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:12.496294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:12.588198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:12.593783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfaed12-7848-4ac1-a3d9-1724ed6ef786 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:12.627987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:12.628559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:12.628867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:12.629141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:12.629861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:12.630168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:12.631565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:12.642636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 610/1243] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:03:12.648866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:12.728964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:12.749889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1acddeda-8a68-46ed-819d-cd4eae86e900 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:12.783808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:12.784590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:12.784871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:12.785144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:12.791586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:12.793022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:03:12.807631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:12.813546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:12.814467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 634/1244] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:03:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:12.815610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:12.914250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2dea8-ac98-467f-bc08-0accbd5d9c38 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:12.947018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f71f5a-b08d-4e8e-a9a0-6f42a9759154 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:12.947666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f71f5a-b08d-4e8e-a9a0-6f42a9759154 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:12.947943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f71f5a-b08d-4e8e-a9a0-6f42a9759154 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:12.948214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f71f5a-b08d-4e8e-a9a0-6f42a9759154 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:12.949037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f71f5a-b08d-4e8e-a9a0-6f42a9759154 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:12.949350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f71f5a-b08d-4e8e-a9a0-6f42a9759154 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:12.950809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26f71f5a-b08d-4e8e-a9a0-6f42a9759154 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:12.963666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 611/1245] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:03:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:03:12.969776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:13.068378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b38b26bc-11f3-46d7-85ac-7a3874bc4755 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:13.101445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47cc944-78bb-47c3-8b8e-a58e069d9090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:13.102103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47cc944-78bb-47c3-8b8e-a58e069d9090 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:13.102388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47cc944-78bb-47c3-8b8e-a58e069d9090 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:13.102726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47cc944-78bb-47c3-8b8e-a58e069d9090 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:13.103588 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47cc944-78bb-47c3-8b8e-a58e069d9090 None admin] RBAC: Adding query filter params (role.id=0fd45f27e71d455da928a71546fbe85e, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=a9c50ae1ba494c20b6133a96dfba21b3) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:13.103902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47cc944-78bb-47c3-8b8e-a58e069d9090 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:13.105325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47cc944-78bb-47c3-8b8e-a58e069d9090 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:13.111088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 635/1246] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:03:12 2022] GET /identity/v3/role_assignments?user.id=a9c50ae1ba494c20b6133a96dfba21b3&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=0fd45f27e71d455da928a71546fbe85e => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:15.194750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165b40d9-a302-4a72-aaaa-d4cee64a2c93 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:15.195315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165b40d9-a302-4a72-aaaa-d4cee64a2c93 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:15.195439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165b40d9-a302-4a72-aaaa-d4cee64a2c93 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:15.196728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 612/1247] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:15.201965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eaf7fb-bba4-415e-b756-2e26a9c58752 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:15.204402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eaf7fb-bba4-415e-b756-2e26a9c58752 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:15.204402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1eaf7fb-bba4-415e-b756-2e26a9c58752 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:15.208215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:15.220024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e1eaf7fb-bba4-415e-b756-2e26a9c58752 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:15.221235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e1eaf7fb-bba4-415e-b756-2e26a9c58752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:15.285804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 636/1248] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:15 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 14 12:03:15.309733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165b40d9-a302-4a72-aaaa-d4cee64a2c93 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:15.328551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:15.408333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165b40d9-a302-4a72-aaaa-d4cee64a2c93 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:15.441462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:15.442250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:15.442533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:15.442834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:15.456394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:03:15.461942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:15.463086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:15.464125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 613/1249] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:03:15 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:15.469292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eaf7fb-bba4-415e-b756-2e26a9c58752 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:15.566158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1eaf7fb-bba4-415e-b756-2e26a9c58752 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:15.595675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:15.601138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:15.601807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:15.602093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:15.602373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:15.603150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:15.603464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:15.604834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:15.612894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 637/1250] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:03:15 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:15.618821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:15.697187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:15.727894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cb8b4c-8f9c-4bdf-8399-e81de072e845 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:15.753616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:15.754398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:15.754684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:15.754988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:15.766075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:15.767566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:15.786118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:15.787273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:15.788186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:15.788186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:15.788186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:15.788186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:15.788186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 614/1251] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:15.793187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:15.893737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a90e933a-711d-4a37-880e-b54a59fc4136 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:15.925920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:15.926568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:15.926887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:15.927165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:15.927877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:15.928184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:15.929550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:15.941724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 638/1252] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:15.947556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:16.049211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3d7e7c-c0dd-4844-b761-dc043fbda869 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:16.075475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:16.083183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:16.083975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:16.084257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:16.084522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:16.092059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:16.093531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:03:16.107462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:16.108586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:16.109662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 615/1253] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:15 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:16.115118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:16.170787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:16.214498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39299438-3e10-43f5-ba5c-abbd29db35d3 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:16.249066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:16.249732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:16.250017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:16.250293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:16.251162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:16.251497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:16.253780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:16.270776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 639/1254] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Jul 14 12:03:16 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:16.276534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:16.376567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba4d9804-a83e-4f33-b1af-3b5d0d74b9d7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:16.409902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:16.410553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:16.410863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:16.411141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:16.465389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:16.477951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:16.479468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:16.576551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 616/1255] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:03:16 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:03:16.582373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:16.681551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7b1005f-f6fa-4b5f-afee-a289d25e5f07 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:16.695141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:16.716215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:16.717012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:16.717295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:16.717570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:16.728271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:16.729767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:16.747571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:16.748554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:16.749610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:16.749610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:16.749610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:16.749610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:16.749610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 640/1256] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:16.754552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:16.841424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:16.853746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a6302b-836d-42cc-b512-d9c76991b0ec None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:16.886193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:16.886878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:16.887165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:16.887444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:16.888164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:16.888476 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:16.889834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:16.901334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 617/1257] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:16.907329 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:17.006580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aa2c94-310d-4f6b-bec5-a638dc7ada64 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:17.041236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:17.042115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:17.042394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:17.042666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:17.049114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:17.051489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:03:17.064769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:17.066523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:17.071639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 641/1258] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:03:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:17.072681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:17.170780 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3cdd64-2645-43be-89ab-a46c3772b337 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:17.203970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6855bdd0-f942-4ca3-a825-7026c70b2352 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:17.204614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855bdd0-f942-4ca3-a825-7026c70b2352 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:17.205004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855bdd0-f942-4ca3-a825-7026c70b2352 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:17.205285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6855bdd0-f942-4ca3-a825-7026c70b2352 None admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:17.206132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6855bdd0-f942-4ca3-a825-7026c70b2352 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:17.206442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6855bdd0-f942-4ca3-a825-7026c70b2352 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:17.207904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6855bdd0-f942-4ca3-a825-7026c70b2352 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:17.219704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 618/1259] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:03:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:17.226274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:17.252608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:17.326814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a726cb93-2041-4faa-a67e-add324254330 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:17.361540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756d4c3a-f19a-45bd-97b8-241234411023 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:17.362291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d4c3a-f19a-45bd-97b8-241234411023 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:17.363181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d4c3a-f19a-45bd-97b8-241234411023 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:17.363459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d4c3a-f19a-45bd-97b8-241234411023 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:17.364320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756d4c3a-f19a-45bd-97b8-241234411023 None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=a9c50ae1ba494c20b6133a96dfba21b3) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:17.364634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756d4c3a-f19a-45bd-97b8-241234411023 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:17.366209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756d4c3a-f19a-45bd-97b8-241234411023 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:17.371904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 642/1260] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:03:17 2022] GET /identity/v3/role_assignments?user.id=a9c50ae1ba494c20b6133a96dfba21b3&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 259 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:19.375189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccc043d-6d98-4561-9a4f-15e72c960caf None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:19.375811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccc043d-6d98-4561-9a4f-15e72c960caf None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:19.375811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bccc043d-6d98-4561-9a4f-15e72c960caf None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:19.377188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 619/1261] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:19.382591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec93c26-9ae9-4d52-bdfe-8e85e23bd1f2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:19.383167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec93c26-9ae9-4d52-bdfe-8e85e23bd1f2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:19.383275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec93c26-9ae9-4d52-bdfe-8e85e23bd1f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:19.400847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-6ec93c26-9ae9-4d52-bdfe-8e85e23bd1f2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:19.402029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6ec93c26-9ae9-4d52-bdfe-8e85e23bd1f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:19.469189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 643/1262] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:19 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:19.493118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccc043d-6d98-4561-9a4f-15e72c960caf None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:19.504187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:19.595989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bccc043d-6d98-4561-9a4f-15e72c960caf None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:19.629851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:19.630575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:19.630884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:19.631162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:19.641115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:03:19.649029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:19.650144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:19.651079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 620/1263] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:03:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:19.656569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec93c26-9ae9-4d52-bdfe-8e85e23bd1f2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:19.676698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:19.756930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec93c26-9ae9-4d52-bdfe-8e85e23bd1f2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:19.790840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:19.797824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:19.797824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:19.797824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:19.797824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:19.797824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:19.797824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:19.803099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 644/1264] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:03:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:19.810050 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:19.876044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:19.908411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8b68f2-a941-4e12-989d-c96eb74d5c3f None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:19.940553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:19.941491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:19.941770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:19.942043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:19.952757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:19.954194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:19.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:19.978722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:19.979667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:19.979667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:19.979667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:19.979667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:19.979667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 621/1265] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:19.981220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:20.067974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:20.086368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96768cd9-a2a2-474e-a678-62951d711f0c None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:20.119339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:20.119995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:20.120296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:20.120556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:20.121272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:20.121580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:20.122983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:20.135045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 645/1266] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:20.140786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:20.241308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b13e42e9-277d-46a6-9cd3-5a1afd5a8e51 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:20.260412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:20.274024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:20.274932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:20.275186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:20.275464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:20.283100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:20.285105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:03:20.299894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:20.304626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:20.305585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 622/1267] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:20 2022] GET /identity/v3/users/placement => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:20.307203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:20.407404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef32daf-35a6-43bf-81d1-2708126b6acd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:20.441371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:20.442034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:20.442315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:20.442592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:20.443452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:20.443763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:20.445191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:20.454216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:20.462598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 646/1268] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Jul 14 12:03:20 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:20.468208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:20.569851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff331c1-b7b6-4057-b1d6-d61b81bc95f8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:20.603980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:20.604809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:20.605093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:20.605367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:20.616622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:20.618067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:20.634612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:20.636343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:20.637526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:20.637526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:20.637526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:20.637526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:20.637526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 623/1269] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:03:20.641549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:20.740376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83435d05-155a-494d-bb34-8cd046890eb5 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:20.773895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:20.774546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:20.774855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:20.775128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:20.775885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:20.776190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:20.777545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:20.789369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 647/1270] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:20.795167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:20.826298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:20.891801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fc82c5-b8da-47d7-94be-b81bdaeaeb45 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:20.925707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:20.926497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:20.926800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:20.927080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:20.933832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:20.935933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:03:20.950576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:20.951623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:20.952532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 624/1271] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:03:20 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:20.958319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:21.015877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:21.059162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205ac3ce-47bd-4a24-9baa-4f101689d171 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:21.092383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a3570b-0bb6-4a77-b85e-3ea5e7d1e27c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:21.092978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a3570b-0bb6-4a77-b85e-3ea5e7d1e27c None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:21.093258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a3570b-0bb6-4a77-b85e-3ea5e7d1e27c None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:21.093536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a3570b-0bb6-4a77-b85e-3ea5e7d1e27c None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:21.094310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a3570b-0bb6-4a77-b85e-3ea5e7d1e27c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:21.094618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a3570b-0bb6-4a77-b85e-3ea5e7d1e27c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:21.096131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a3570b-0bb6-4a77-b85e-3ea5e7d1e27c None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:21.107603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 648/1272] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:03:20 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:21.113780 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:21.208929 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:21.217388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ae070d-3bf6-4847-a020-3a821f120486 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:21.250249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27982c99-ebbc-45d0-b77b-b3aff9979326 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:21.251185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27982c99-ebbc-45d0-b77b-b3aff9979326 None admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:21.251600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27982c99-ebbc-45d0-b77b-b3aff9979326 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:21.251960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27982c99-ebbc-45d0-b77b-b3aff9979326 None admin] PATH_INFO: `/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/a9c50ae1ba494c20b6133a96dfba21b3/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:21.254960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27982c99-ebbc-45d0-b77b-b3aff9979326 None admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d22dbd90a47539c4e594aee69a482, user_id=a9c50ae1ba494c20b6133a96dfba21b3, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:21.256521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27982c99-ebbc-45d0-b77b-b3aff9979326 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:21.395596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 625/1273] 10.222.0.27 () {58 vars in 1439 bytes} [Thu Jul 14 12:03:21 2022] PUT /identity/v3/projects/aa1d22dbd90a47539c4e594aee69a482/users/a9c50ae1ba494c20b6133a96dfba21b3/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:03:23.418289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d8c4fc-b37f-4680-bd0a-81c7f3cf11a3 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:23.418968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d8c4fc-b37f-4680-bd0a-81c7f3cf11a3 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:23.418968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d8c4fc-b37f-4680-bd0a-81c7f3cf11a3 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:23.420253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 649/1274] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:23.426170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aff261-b6ca-4a36-a217-a4353fa0ad48 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:23.426614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aff261-b6ca-4a36-a217-a4353fa0ad48 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:23.426728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55aff261-b6ca-4a36-a217-a4353fa0ad48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:23.443883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-55aff261-b6ca-4a36-a217-a4353fa0ad48 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:23.445386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-55aff261-b6ca-4a36-a217-a4353fa0ad48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:23.511582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 626/1275] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:23.536359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d8c4fc-b37f-4680-bd0a-81c7f3cf11a3 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:23.637183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d8c4fc-b37f-4680-bd0a-81c7f3cf11a3 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:23.655854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:23.674030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:23.674805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:23.675099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:23.675375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:23.685276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 14 12:03:23.692999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:23.697281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:23.698353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 650/1276] 10.222.0.27 () {56 vars in 1096 bytes} [Thu Jul 14 12:03:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:23.700293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aff261-b6ca-4a36-a217-a4353fa0ad48 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:23.782682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:23.800799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55aff261-b6ca-4a36-a217-a4353fa0ad48 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:23.833726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:23.834411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:23.834740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:23.835238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:23.836034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:23.836631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:23.839923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:23.850014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 627/1277] 10.222.0.27 () {56 vars in 1099 bytes} [Thu Jul 14 12:03:23 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:23.855661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:23.952433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37860494-9023-4e12-bb69-dc77a9085a80 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:23.984662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:23.985450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:23.985726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:23.985998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:23.998265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:23.999743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:24.017070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:24.020466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:24.021629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:24.021629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:24.021629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:24.021629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:24.021629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 651/1278] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:24.024899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:24.123883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dff1426-009a-4a82-8b0a-7b3e9904d8a8 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:24.159886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:24.160585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:24.160870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:24.161150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:24.161874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:24.162184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:24.163588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:24.176618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 628/1279] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:24.182294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:24.213825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:24.281719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8299e5-fa0d-485d-9aca-de6c393b5260 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:24.316891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:24.317686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:24.317969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:24.318270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] PATH_INFO: `/v3/users/placement` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:24.325238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:24.326748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 14 12:03:24.342196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:24.343185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:24.344148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 652/1280] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:24 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 14 12:03:24.349477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:24.353305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:24.450713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e5d310-e491-43cd-bb0d-f4cd80af5d19 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:24.484464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:24.485073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:24.485421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:24.485664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:24.486499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:24.487349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:24.488827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:24.506012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 629/1281] 10.222.0.27 () {56 vars in 1143 bytes} [Thu Jul 14 12:03:24 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:24.511845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:24.600891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:24.610133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a224dc-c49f-4d82-a44a-5582ff0854aa None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:24.645953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:24.646814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:24.647093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:24.647552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:24.719442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:24.721141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:24.808079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:24.818608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 653/1282] 10.222.0.27 () {58 vars in 1282 bytes} [Thu Jul 14 12:03:24 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 14 12:03:24.824479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:24.885112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:24.923323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe86de96-a500-4722-b26b-4a5da4b6fed7 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:24.958282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:24.959125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:24.959412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:24.959690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:24.969653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:24.971501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 14 12:03:24.989193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:24.990222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:24.991448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 14 12:03:24.991448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 14 12:03:24.991448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 14 12:03:24.991448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:24.991448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 630/1283] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:03:24 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:24.996201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:25.099437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66472e0f-5196-4db2-bced-90c8a6a817c0 None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:25.132237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:25.132892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:25.133170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:25.133443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:25.134152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:25.134456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:25.135857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:25.147562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 654/1284] 10.222.0.27 () {56 vars in 1109 bytes} [Thu Jul 14 12:03:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:25.153381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:25.252521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f0b506c-a759-42c9-a011-b9e2f4cf6262 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:25.266288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:25.287221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:25.288047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:25.288332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:25.288608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:25.295165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:25.296654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 14 12:03:25.310989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:25.311899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: ERROR keystone.server.flask.application Jul 14 12:03:25.312791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 631/1285] 10.222.0.27 () {56 vars in 1111 bytes} [Thu Jul 14 12:03:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:25.318306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:25.373671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:25.424036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1345757-3398-468c-9e28-e0ca52e5db6b None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:25.458373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4931c0cc-79fd-484a-9a43-3803afb6e812 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:25.459060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931c0cc-79fd-484a-9a43-3803afb6e812 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:25.459346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931c0cc-79fd-484a-9a43-3803afb6e812 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:25.459624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931c0cc-79fd-484a-9a43-3803afb6e812 None admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:25.460411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4931c0cc-79fd-484a-9a43-3803afb6e812 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:25.460725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4931c0cc-79fd-484a-9a43-3803afb6e812 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:25.462213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4931c0cc-79fd-484a-9a43-3803afb6e812 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:25.473934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 655/1286] 10.222.0.27 () {56 vars in 1148 bytes} [Thu Jul 14 12:03:25 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:25.480087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:25.579301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77c5e59-4692-4df9-aa49-12affac7e952 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:25.613657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5e5626-61bd-4018-9703-edfd6f732c1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:25.614301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5e5626-61bd-4018-9703-edfd6f732c1b None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:25.614585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5e5626-61bd-4018-9703-edfd6f732c1b None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:25.614897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5e5626-61bd-4018-9703-edfd6f732c1b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:25.615761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5e5626-61bd-4018-9703-edfd6f732c1b None admin] RBAC: Adding query filter params (role.id=e1d0ea88764c4c89a1159e9b09076352, scope.project.id=aa1d22dbd90a47539c4e594aee69a482, user.id=a9c50ae1ba494c20b6133a96dfba21b3) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:25.616079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5e5626-61bd-4018-9703-edfd6f732c1b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:25.617532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5e5626-61bd-4018-9703-edfd6f732c1b None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:25.623488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 632/1287] 10.222.0.27 () {56 vars in 1374 bytes} [Thu Jul 14 12:03:25 2022] GET /identity/v3/role_assignments?user.id=a9c50ae1ba494c20b6133a96dfba21b3&scope.project.id=aa1d22dbd90a47539c4e594aee69a482&role.id=e1d0ea88764c4c89a1159e9b09076352 => generated 608 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:27.685686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2793a12f-7350-4071-80bf-e26098af16ae None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:27.686249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2793a12f-7350-4071-80bf-e26098af16ae None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:27.686249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2793a12f-7350-4071-80bf-e26098af16ae None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:27.687645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 656/1288] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:27.693488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae3185-66bb-4055-acce-ae1bc5fee98f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:27.695533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae3185-66bb-4055-acce-ae1bc5fee98f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:27.695533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daae3185-66bb-4055-acce-ae1bc5fee98f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:27.712457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-daae3185-66bb-4055-acce-ae1bc5fee98f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:27.713671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-daae3185-66bb-4055-acce-ae1bc5fee98f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:27.743735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:27.780541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 633/1289] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:27 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:27.803089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2793a12f-7350-4071-80bf-e26098af16ae None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:27.904469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2793a12f-7350-4071-80bf-e26098af16ae None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:27.914956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:27.937967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:27.938700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:27.939019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:27.939297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] PATH_INFO: `/v3/services/placement` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:27.940471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:27.941882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:27.953659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:27.954728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:27.957156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 657/1290] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:03:27 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:27.961577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daae3185-66bb-4055-acce-ae1bc5fee98f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:28.059235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daae3185-66bb-4055-acce-ae1bc5fee98f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:28.092318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd80f7c1-47e5-4386-80d1-1a1a78152932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:28.092976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd80f7c1-47e5-4386-80d1-1a1a78152932 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:28.093260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd80f7c1-47e5-4386-80d1-1a1a78152932 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:28.093537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd80f7c1-47e5-4386-80d1-1a1a78152932 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:28.094244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd80f7c1-47e5-4386-80d1-1a1a78152932 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:28.094582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd80f7c1-47e5-4386-80d1-1a1a78152932 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:28.096066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd80f7c1-47e5-4386-80d1-1a1a78152932 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:28.100813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 634/1291] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Jul 14 12:03:27 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:28.106514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:28.205822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08b2510a-867a-4fe0-b80a-8ac0af890fcf None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:28.238271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a7d760-dba8-487d-8396-c0a77dccece9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:28.238950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a7d760-dba8-487d-8396-c0a77dccece9 None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:28.239232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a7d760-dba8-487d-8396-c0a77dccece9 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:28.239507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a7d760-dba8-487d-8396-c0a77dccece9 None admin] PATH_INFO: `/v3/services` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:28.240204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a7d760-dba8-487d-8396-c0a77dccece9 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:28.240509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a7d760-dba8-487d-8396-c0a77dccece9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:28.241878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a7d760-dba8-487d-8396-c0a77dccece9 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:28.246284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 658/1292] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Jul 14 12:03:28 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:30.262327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8dcf7-e22a-49b5-a7be-f2e845e8ded1 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:30.262885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8dcf7-e22a-49b5-a7be-f2e845e8ded1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:30.262982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5d8dcf7-e22a-49b5-a7be-f2e845e8ded1 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:30.264418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 635/1293] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:30.270406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ff16b-7076-4922-8063-9fcd04decb43 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:30.270847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ff16b-7076-4922-8063-9fcd04decb43 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:30.270999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29ff16b-7076-4922-8063-9fcd04decb43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:30.287904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d29ff16b-7076-4922-8063-9fcd04decb43 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:30.289082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d29ff16b-7076-4922-8063-9fcd04decb43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:30.318612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:30.355360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 659/1294] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:30 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:30.379663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d8dcf7-e22a-49b5-a7be-f2e845e8ded1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:30.410991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:30.480432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5d8dcf7-e22a-49b5-a7be-f2e845e8ded1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:30.514828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd37c27f-5f54-476f-baa4-e0526b3d66c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:30.515418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd37c27f-5f54-476f-baa4-e0526b3d66c4 None admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:30.515736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd37c27f-5f54-476f-baa4-e0526b3d66c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:30.516022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd37c27f-5f54-476f-baa4-e0526b3d66c4 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:30.516752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd37c27f-5f54-476f-baa4-e0526b3d66c4 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:30.518212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd37c27f-5f54-476f-baa4-e0526b3d66c4 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:30.611384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 636/1295] 10.222.0.27 () {60 vars in 1141 bytes} [Thu Jul 14 12:03:30 2022] POST /identity/v3/services => generated 246 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:03:32.651827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a669a9a9-7017-4ec8-98c3-f08f0abcacdd None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:32.652375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a669a9a9-7017-4ec8-98c3-f08f0abcacdd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:32.652375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a669a9a9-7017-4ec8-98c3-f08f0abcacdd None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:32.653746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 660/1296] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:32.659621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228d91cb-6e13-486e-a4b4-9efb763c4864 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:32.660074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228d91cb-6e13-486e-a4b4-9efb763c4864 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:32.660166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228d91cb-6e13-486e-a4b4-9efb763c4864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:32.676623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-228d91cb-6e13-486e-a4b4-9efb763c4864 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:32.677815 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-228d91cb-6e13-486e-a4b4-9efb763c4864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:32.749878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 637/1297] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:32 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:32.773114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a669a9a9-7017-4ec8-98c3-f08f0abcacdd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:32.872728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a669a9a9-7017-4ec8-98c3-f08f0abcacdd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:32.906967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:32.907377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:32.907659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:32.907929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] PATH_INFO: `/v3/services/placement` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:32.908636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:32.910352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:32.922167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:32.923371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:32.925085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 661/1298] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:03:32 2022] GET /identity/v3/services/placement => generated 90 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:32.930081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228d91cb-6e13-486e-a4b4-9efb763c4864 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:32.949563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:33.028072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228d91cb-6e13-486e-a4b4-9efb763c4864 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:33.061605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:33.062266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:33.062555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:33.062867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:33.063658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:33.063968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:33.065471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:33.070616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 638/1299] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Jul 14 12:03:32 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:33.076563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:33.111013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:33.178214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b84dee-b7b2-439e-821a-564060fe08cd None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:33.211536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d58416b-6fad-4ba4-8f6b-f5977b0402bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:33.218708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d58416b-6fad-4ba4-8f6b-f5977b0402bb None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:33.218708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d58416b-6fad-4ba4-8f6b-f5977b0402bb None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:33.218708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d58416b-6fad-4ba4-8f6b-f5977b0402bb None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:33.218708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d58416b-6fad-4ba4-8f6b-f5977b0402bb None admin] RBAC: Adding query filter params (interface=public, service_id=f2259a72ed4c4d7ea51813c3b5b348f7, region_id=RegionOne) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:33.218708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d58416b-6fad-4ba4-8f6b-f5977b0402bb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:33.218708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d58416b-6fad-4ba4-8f6b-f5977b0402bb None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:33.220200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 662/1300] 10.222.0.27 () {56 vars in 1251 bytes} [Thu Jul 14 12:03:33 2022] GET /identity/v3/endpoints?service_id=f2259a72ed4c4d7ea51813c3b5b348f7&interface=public®ion_id=RegionOne => generated 194 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:33.226284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:33.306825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:33.324115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d393882c-f7cc-41cd-a258-14817e12d886 None admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:33.358266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86593c98-6e4f-4b95-8c0a-957fc227b017 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:33.359016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86593c98-6e4f-4b95-8c0a-957fc227b017 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:33.359245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86593c98-6e4f-4b95-8c0a-957fc227b017 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:33.359513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86593c98-6e4f-4b95-8c0a-957fc227b017 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:33.360209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86593c98-6e4f-4b95-8c0a-957fc227b017 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:33.361643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86593c98-6e4f-4b95-8c0a-957fc227b017 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:33.368552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 639/1301] 10.222.0.27 () {56 vars in 1087 bytes} [Thu Jul 14 12:03:33 2022] GET /identity/v3/services => generated 2191 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:03:35.394066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad3621-dc1b-4648-b96b-a0d13100e47d None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:35.394628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad3621-dc1b-4648-b96b-a0d13100e47d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:35.394748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ad3621-dc1b-4648-b96b-a0d13100e47d None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:35.396179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 663/1302] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:03:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:03:35.401310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92ca2dc-0ef5-43b8-a9d6-f0d379a9e0ed None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:35.401752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92ca2dc-0ef5-43b8-a9d6-f0d379a9e0ed None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:35.401898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92ca2dc-0ef5-43b8-a9d6-f0d379a9e0ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:35.418523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b92ca2dc-0ef5-43b8-a9d6-f0d379a9e0ed None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:03:35.419752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b92ca2dc-0ef5-43b8-a9d6-f0d379a9e0ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:03:35.486511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 640/1303] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:03:35 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 14 12:03:35.509764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad3621-dc1b-4648-b96b-a0d13100e47d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:35.603312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:35.610086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ad3621-dc1b-4648-b96b-a0d13100e47d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:35.643811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:35.644631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:35.644858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:35.645127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] PATH_INFO: `/v3/services/placement` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:35.645770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:35.647150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 14 12:03:35.659349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 14 12:03:35.664707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.get_or_create( Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application with Lock( Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self._enter() Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application return self.creator() Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application created_value = creator( Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: ERROR keystone.server.flask.application Jul 14 12:03:35.665614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 664/1304] 10.222.0.27 () {56 vars in 1117 bytes} [Thu Jul 14 12:03:35 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 14 12:03:35.666696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92ca2dc-0ef5-43b8-a9d6-f0d379a9e0ed None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:35.744354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:35.767104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92ca2dc-0ef5-43b8-a9d6-f0d379a9e0ed None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:35.800317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e38344b4-fe8a-41ed-8453-930cc1fbd995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:35.800985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38344b4-fe8a-41ed-8453-930cc1fbd995 None admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:35.801274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38344b4-fe8a-41ed-8453-930cc1fbd995 None admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:35.801553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e38344b4-fe8a-41ed-8453-930cc1fbd995 None admin] PATH_INFO: `/v3/services` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:35.802257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38344b4-fe8a-41ed-8453-930cc1fbd995 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:03:35.802637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38344b4-fe8a-41ed-8453-930cc1fbd995 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:35.804073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e38344b4-fe8a-41ed-8453-930cc1fbd995 None admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:35.809219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 641/1305] 10.222.0.27 () {56 vars in 1116 bytes} [Thu Jul 14 12:03:35 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:03:35.816180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:03:35.915564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af5eeeb-d01f-473f-9d54-63be3f68c05f None admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:03:35.950238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1dd823af-d40a-4a12-b450-765593cae542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:03:35.950946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd823af-d40a-4a12-b450-765593cae542 None admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:03:35.951228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd823af-d40a-4a12-b450-765593cae542 None admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:03:35.951506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dd823af-d40a-4a12-b450-765593cae542 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:03:35.952246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd823af-d40a-4a12-b450-765593cae542 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:03:35.953642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1dd823af-d40a-4a12-b450-765593cae542 None admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:03:36.035017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 665/1306] 10.222.0.27 () {60 vars in 1144 bytes} [Thu Jul 14 12:03:35 2022] POST /identity/v3/endpoints => generated 331 bytes in 220 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:03:45.812544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=30 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:03:46.029852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:40.964105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb242b24-2324-4235-bcae-e3f7f360cf24 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:40.964683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb242b24-2324-4235-bcae-e3f7f360cf24 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:40.964683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb242b24-2324-4235-bcae-e3f7f360cf24 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:40.971005 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4348e-2c72-4ffb-8fb8-1be18ad704d1 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:40.972098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4348e-2c72-4ffb-8fb8-1be18ad704d1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:40.972233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4348e-2c72-4ffb-8fb8-1be18ad704d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:40.975062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 642/1307] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:09:40 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:41.054198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-71c4348e-2c72-4ffb-8fb8-1be18ad704d1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:41.055532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-71c4348e-2c72-4ffb-8fb8-1be18ad704d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:41.179726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 666/1308] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:09:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:09:41.236762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc5fc6-7ecb-48f1-ba91-c1d8ecc05bd4 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:41.237085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc5fc6-7ecb-48f1-ba91-c1d8ecc05bd4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:41.237450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcc5fc6-7ecb-48f1-ba91-c1d8ecc05bd4 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:41.239265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 643/1309] 10.222.0.27 () {54 vars in 964 bytes} [Thu Jul 14 12:09:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:41.250023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af4335-8396-486a-b2f1-a18ac42d844d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:41.250317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af4335-8396-486a-b2f1-a18ac42d844d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:41.250597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67af4335-8396-486a-b2f1-a18ac42d844d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:41.378651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-67af4335-8396-486a-b2f1-a18ac42d844d None None] MFA Rules not processed for user `a4fff90ae5434b58932f468ca9c827f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:41.382015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-67af4335-8396-486a-b2f1-a18ac42d844d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:41.431805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:41.522170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 667/1310] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Jul 14 12:09:41 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 273 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:09:41.531769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcc5fc6-7ecb-48f1-ba91-c1d8ecc05bd4 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:09:41.534313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9fcc5fc6-7ecb-48f1-ba91-c1d8ecc05bd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:41.660644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcc5fc6-7ecb-48f1-ba91-c1d8ecc05bd4 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:09:41.741040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3f4a65-66a4-499d-b315-129469c336ac service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:09:41.741728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f4a65-66a4-499d-b315-129469c336ac service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:41.742261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f4a65-66a4-499d-b315-129469c336ac service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:41.742539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3f4a65-66a4-499d-b315-129469c336ac service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:41.744227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ca3f4a65-66a4-499d-b315-129469c336ac service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:41.799276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:41.856688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3f4a65-66a4-499d-b315-129469c336ac service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:09:41.858154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3f4a65-66a4-499d-b315-129469c336ac service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:09:41.879213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 644/1311] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:09:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 347 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:09:44.212275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986fff8e-27a1-4e9f-882f-92f99591ac5d None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:44.212835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986fff8e-27a1-4e9f-882f-92f99591ac5d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:44.212835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986fff8e-27a1-4e9f-882f-92f99591ac5d None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:44.214204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 668/1312] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:09:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:44.219414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5d2df-0a7c-414f-a3d1-e4b7a240e68e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:44.219878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5d2df-0a7c-414f-a3d1-e4b7a240e68e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:44.220042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a5d2df-0a7c-414f-a3d1-e4b7a240e68e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:44.309167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-69a5d2df-0a7c-414f-a3d1-e4b7a240e68e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:44.310410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-69a5d2df-0a7c-414f-a3d1-e4b7a240e68e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:44.415575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 645/1313] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:09:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:09:44.453788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986fff8e-27a1-4e9f-882f-92f99591ac5d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:09:44.455354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-986fff8e-27a1-4e9f-882f-92f99591ac5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:44.559467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986fff8e-27a1-4e9f-882f-92f99591ac5d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:09:44.632620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:44.635171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a4613e-92e0-4adb-9f5d-e7e7c2ce2c65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:09:44.635814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4613e-92e0-4adb-9f5d-e7e7c2ce2c65 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:44.636094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4613e-92e0-4adb-9f5d-e7e7c2ce2c65 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:44.636366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a4613e-92e0-4adb-9f5d-e7e7c2ce2c65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:44.638007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c1a4613e-92e0-4adb-9f5d-e7e7c2ce2c65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:44.754164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a4613e-92e0-4adb-9f5d-e7e7c2ce2c65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:09:44.755619 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a4613e-92e0-4adb-9f5d-e7e7c2ce2c65 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:09:44.773678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 669/1314] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:09:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:09:47.345939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caceefb-d4a3-4b4a-abc7-be2bab0f9b9e None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:47.346621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caceefb-d4a3-4b4a-abc7-be2bab0f9b9e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:47.346621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0caceefb-d4a3-4b4a-abc7-be2bab0f9b9e None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:47.348072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 646/1315] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:09:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:47.353137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274a9b-976d-4b01-bd5a-fb11fadce878 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:47.353581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274a9b-976d-4b01-bd5a-fb11fadce878 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:47.353696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88274a9b-976d-4b01-bd5a-fb11fadce878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:47.420659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-88274a9b-976d-4b01-bd5a-fb11fadce878 None None] MFA Rules not processed for user `56dd0fcc7125401f8bc4841e48ca9167`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:47.421933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-88274a9b-976d-4b01-bd5a-fb11fadce878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:47.473511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:47.517050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 670/1316] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:09:47 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:09:47.569454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6376dc70-2b2a-47e5-bfd0-5f8b8728f41b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:47.569649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6376dc70-2b2a-47e5-bfd0-5f8b8728f41b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:47.569925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6376dc70-2b2a-47e5-bfd0-5f8b8728f41b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:47.571346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 647/1317] 10.222.0.27 () {54 vars in 964 bytes} [Thu Jul 14 12:09:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:47.577142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141d1fd8-ef94-4c20-ae43-05e8842c9426 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:47.577429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141d1fd8-ef94-4c20-ae43-05e8842c9426 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:47.577706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141d1fd8-ef94-4c20-ae43-05e8842c9426 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:47.651202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-141d1fd8-ef94-4c20-ae43-05e8842c9426 None None] MFA Rules not processed for user `a4fff90ae5434b58932f468ca9c827f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:47.652402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-141d1fd8-ef94-4c20-ae43-05e8842c9426 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:47.725926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 671/1318] 10.222.0.27 () {58 vars in 1061 bytes} [Thu Jul 14 12:09:47 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:09:47.734746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6376dc70-2b2a-47e5-bfd0-5f8b8728f41b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:09:47.736321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6376dc70-2b2a-47e5-bfd0-5f8b8728f41b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:47.746858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:47.838495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6376dc70-2b2a-47e5-bfd0-5f8b8728f41b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:09:47.909844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7739c88-40a9-4353-b5cf-32fb4b7269b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:09:47.910473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7739c88-40a9-4353-b5cf-32fb4b7269b7 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:47.910753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7739c88-40a9-4353-b5cf-32fb4b7269b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:47.911069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7739c88-40a9-4353-b5cf-32fb4b7269b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:47.912690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c7739c88-40a9-4353-b5cf-32fb4b7269b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:47.973682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:48.018414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7739c88-40a9-4353-b5cf-32fb4b7269b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:09:48.019977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7739c88-40a9-4353-b5cf-32fb4b7269b7 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:09:48.038395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 648/1319] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:09:47 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:09:51.520425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7ba63-2585-4d5f-9e0f-1985c00f71b6 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:51.521027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7ba63-2585-4d5f-9e0f-1985c00f71b6 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:51.521027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f7ba63-2585-4d5f-9e0f-1985c00f71b6 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:51.522359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 672/1320] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:09:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:51.527469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9cd4c-a073-4307-9de5-09edbf66fb20 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:51.527871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9cd4c-a073-4307-9de5-09edbf66fb20 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:51.528040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b9cd4c-a073-4307-9de5-09edbf66fb20 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:51.592907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-76b9cd4c-a073-4307-9de5-09edbf66fb20 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:51.594366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-76b9cd4c-a073-4307-9de5-09edbf66fb20 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:51.689178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 649/1321] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:09:51 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:09:51.728869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f7ba63-2585-4d5f-9e0f-1985c00f71b6 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:09:51.730406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e6f7ba63-2585-4d5f-9e0f-1985c00f71b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:51.780125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:51.834309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f7ba63-2585-4d5f-9e0f-1985c00f71b6 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:09:51.904400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75b1dcb-0c43-4c48-83f7-94bdbd49b267 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:09:51.905035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b1dcb-0c43-4c48-83f7-94bdbd49b267 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:51.905311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b1dcb-0c43-4c48-83f7-94bdbd49b267 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:51.905579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75b1dcb-0c43-4c48-83f7-94bdbd49b267 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:51.907493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d75b1dcb-0c43-4c48-83f7-94bdbd49b267 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:51.998590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:52.011969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75b1dcb-0c43-4c48-83f7-94bdbd49b267 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:09:52.013390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75b1dcb-0c43-4c48-83f7-94bdbd49b267 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:09:52.031957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 673/1322] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:09:51 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:09:55.136757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299ffce-3c1e-4ff3-9f13-270fe96b4032 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:55.137330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299ffce-3c1e-4ff3-9f13-270fe96b4032 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:55.137330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f299ffce-3c1e-4ff3-9f13-270fe96b4032 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:55.138717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 650/1323] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:09:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:55.144864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc375dbc-a6c3-437d-b638-67c5aebb2c6d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:55.145283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc375dbc-a6c3-437d-b638-67c5aebb2c6d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:55.145424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc375dbc-a6c3-437d-b638-67c5aebb2c6d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:55.211482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-dc375dbc-a6c3-437d-b638-67c5aebb2c6d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:55.212736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-dc375dbc-a6c3-437d-b638-67c5aebb2c6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:55.311648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 674/1324] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:09:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:09:55.349739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f299ffce-3c1e-4ff3-9f13-270fe96b4032 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:09:55.351445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f299ffce-3c1e-4ff3-9f13-270fe96b4032 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:55.374196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:55.467868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f299ffce-3c1e-4ff3-9f13-270fe96b4032 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:09:55.548002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1a4b0f8-538c-4ed6-bf57-bfedefcf9d94 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:09:55.548647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a4b0f8-538c-4ed6-bf57-bfedefcf9d94 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:55.549008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a4b0f8-538c-4ed6-bf57-bfedefcf9d94 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:55.549779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1a4b0f8-538c-4ed6-bf57-bfedefcf9d94 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:55.551481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e1a4b0f8-538c-4ed6-bf57-bfedefcf9d94 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:55.621812 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:55.660515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a4b0f8-538c-4ed6-bf57-bfedefcf9d94 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:09:55.661959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1a4b0f8-538c-4ed6-bf57-bfedefcf9d94 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:09:55.679491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 651/1325] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:09:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:09:57.766911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f62e83-046e-481f-8be6-4df47b05373d None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:57.767446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f62e83-046e-481f-8be6-4df47b05373d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:57.767446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f62e83-046e-481f-8be6-4df47b05373d None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:57.769246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 675/1326] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:09:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:09:57.774611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f0dc4d-c949-4d22-b3ab-f8a7fac80541 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:57.775092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f0dc4d-c949-4d22-b3ab-f8a7fac80541 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:57.775221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f0dc4d-c949-4d22-b3ab-f8a7fac80541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:57.840947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-55f0dc4d-c949-4d22-b3ab-f8a7fac80541 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:09:57.842069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-55f0dc4d-c949-4d22-b3ab-f8a7fac80541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:57.943543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 652/1327] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:09:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:09:57.979029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f62e83-046e-481f-8be6-4df47b05373d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:09:57.980560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-24f62e83-046e-481f-8be6-4df47b05373d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:58.046401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:58.096386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f62e83-046e-481f-8be6-4df47b05373d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:09:58.169636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddf2397-dff8-4661-910a-db9c970ba71c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:09:58.170313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddf2397-dff8-4661-910a-db9c970ba71c service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:09:58.170595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddf2397-dff8-4661-910a-db9c970ba71c service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:09:58.170903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddf2397-dff8-4661-910a-db9c970ba71c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:09:58.172551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6ddf2397-dff8-4661-910a-db9c970ba71c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:09:58.286183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:09:58.288678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddf2397-dff8-4661-910a-db9c970ba71c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:09:58.290120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddf2397-dff8-4661-910a-db9c970ba71c service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:09:58.308204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 676/1328] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:09:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:00.400328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8892a14-ac9d-4254-9abd-3c711d93807b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:00.400906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8892a14-ac9d-4254-9abd-3c711d93807b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:00.400906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8892a14-ac9d-4254-9abd-3c711d93807b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:00.402290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 653/1329] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:00.408256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77be51a1-7ca1-4ff2-a7d8-e475334a920d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:00.408762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77be51a1-7ca1-4ff2-a7d8-e475334a920d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:00.408954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77be51a1-7ca1-4ff2-a7d8-e475334a920d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:00.475631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-77be51a1-7ca1-4ff2-a7d8-e475334a920d None None] MFA Rules not processed for user `56dd0fcc7125401f8bc4841e48ca9167`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:00.476858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-77be51a1-7ca1-4ff2-a7d8-e475334a920d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:00.577957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 677/1330] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:00 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:00.615137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8892a14-ac9d-4254-9abd-3c711d93807b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:00.616691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e8892a14-ac9d-4254-9abd-3c711d93807b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:00.649408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:00.730721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8892a14-ac9d-4254-9abd-3c711d93807b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:00.807688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ed9acd-6538-4992-88a8-270d2b034bc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:00.808434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ed9acd-6538-4992-88a8-270d2b034bc0 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:00.808719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ed9acd-6538-4992-88a8-270d2b034bc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:00.808999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ed9acd-6538-4992-88a8-270d2b034bc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:00.810643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-56ed9acd-6538-4992-88a8-270d2b034bc0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:00.890072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:00.923146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ed9acd-6538-4992-88a8-270d2b034bc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:00.925269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ed9acd-6538-4992-88a8-270d2b034bc0 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:00.942918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 654/1331] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:00 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:03.416339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724704a9-8384-47bb-aa94-4ccf70ff10df None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:03.416891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724704a9-8384-47bb-aa94-4ccf70ff10df None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:03.416891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-724704a9-8384-47bb-aa94-4ccf70ff10df None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:03.418259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 678/1332] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:03.423369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe78aff-2dd3-4e79-b9ba-83a3a5c03b3c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:03.428971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe78aff-2dd3-4e79-b9ba-83a3a5c03b3c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:03.428971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfe78aff-2dd3-4e79-b9ba-83a3a5c03b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:03.488338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-dfe78aff-2dd3-4e79-b9ba-83a3a5c03b3c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:03.489557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dfe78aff-2dd3-4e79-b9ba-83a3a5c03b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:03.585951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 655/1333] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:03.621323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724704a9-8384-47bb-aa94-4ccf70ff10df None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:03.622880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-724704a9-8384-47bb-aa94-4ccf70ff10df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:03.692028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:03.724707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-724704a9-8384-47bb-aa94-4ccf70ff10df None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:03.797522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20fd18a9-acf4-4183-bdb7-d673d8be9a15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:03.798166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd18a9-acf4-4183-bdb7-d673d8be9a15 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:03.798448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd18a9-acf4-4183-bdb7-d673d8be9a15 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:03.799351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20fd18a9-acf4-4183-bdb7-d673d8be9a15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:03.801058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-20fd18a9-acf4-4183-bdb7-d673d8be9a15 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:03.908814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fd18a9-acf4-4183-bdb7-d673d8be9a15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:03.911033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20fd18a9-acf4-4183-bdb7-d673d8be9a15 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:03.920898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:03.929649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 679/1334] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:05.989373 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2ae5c7-7490-46fc-bd5f-70a9015c346b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:05.989921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2ae5c7-7490-46fc-bd5f-70a9015c346b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:05.989921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2ae5c7-7490-46fc-bd5f-70a9015c346b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:05.991255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 656/1335] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:10:05.996332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f608b-48c2-482e-b61a-9412c08b8050 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:05.996765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f608b-48c2-482e-b61a-9412c08b8050 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:05.996886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f608b-48c2-482e-b61a-9412c08b8050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:06.060788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4c5f608b-48c2-482e-b61a-9412c08b8050 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:06.062002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4c5f608b-48c2-482e-b61a-9412c08b8050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:06.156520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 680/1336] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:10:06.197179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2ae5c7-7490-46fc-bd5f-70a9015c346b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:06.198727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dd2ae5c7-7490-46fc-bd5f-70a9015c346b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:06.239525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:06.301521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2ae5c7-7490-46fc-bd5f-70a9015c346b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:06.373334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e740599-63c9-4005-a541-c8092d6c9933 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:06.373976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e740599-63c9-4005-a541-c8092d6c9933 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:06.374260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e740599-63c9-4005-a541-c8092d6c9933 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:06.374537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e740599-63c9-4005-a541-c8092d6c9933 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:06.376207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9e740599-63c9-4005-a541-c8092d6c9933 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:06.459155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:06.484719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e740599-63c9-4005-a541-c8092d6c9933 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:06.486292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e740599-63c9-4005-a541-c8092d6c9933 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:06.504583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 657/1337] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:09.269316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e7ddf-5dd4-4d03-bd6e-1e72f6811ddf None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:09.269865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e7ddf-5dd4-4d03-bd6e-1e72f6811ddf None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:09.269865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-899e7ddf-5dd4-4d03-bd6e-1e72f6811ddf None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:09.271282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 681/1338] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:09.276332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34794ff-fd2e-4d2f-8609-982f47e4cf9b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:09.276755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34794ff-fd2e-4d2f-8609-982f47e4cf9b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:09.276883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34794ff-fd2e-4d2f-8609-982f47e4cf9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:09.339667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c34794ff-fd2e-4d2f-8609-982f47e4cf9b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:09.340856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c34794ff-fd2e-4d2f-8609-982f47e4cf9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:09.436080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 658/1339] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:09 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:09.472115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e7ddf-5dd4-4d03-bd6e-1e72f6811ddf None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:09.473679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-899e7ddf-5dd4-4d03-bd6e-1e72f6811ddf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:09.551153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:09.578595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-899e7ddf-5dd4-4d03-bd6e-1e72f6811ddf None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:09.652150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a82f8e9-c7e4-448c-9770-eccb6c69e27e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:09.652794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a82f8e9-c7e4-448c-9770-eccb6c69e27e service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:09.653076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a82f8e9-c7e4-448c-9770-eccb6c69e27e service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:09.653352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a82f8e9-c7e4-448c-9770-eccb6c69e27e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:09.655098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9a82f8e9-c7e4-448c-9770-eccb6c69e27e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:09.763215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a82f8e9-c7e4-448c-9770-eccb6c69e27e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:09.764661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a82f8e9-c7e4-448c-9770-eccb6c69e27e service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:09.782773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 682/1340] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:09 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:14.128071 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f48879-1a78-4d7c-8647-6ce0f6a78b60 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:14.128586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f48879-1a78-4d7c-8647-6ce0f6a78b60 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:14.128681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f48879-1a78-4d7c-8647-6ce0f6a78b60 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:14.130168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 659/1341] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:14.135684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da8275c-e7cb-4b94-803b-7f661513506a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:14.136112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da8275c-e7cb-4b94-803b-7f661513506a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:14.136365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da8275c-e7cb-4b94-803b-7f661513506a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:14.145448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:14.209161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8da8275c-e7cb-4b94-803b-7f661513506a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:14.210414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8da8275c-e7cb-4b94-803b-7f661513506a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:14.315248 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 683/1342] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:14.353936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f48879-1a78-4d7c-8647-6ce0f6a78b60 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:14.355609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f6f48879-1a78-4d7c-8647-6ce0f6a78b60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:14.406726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:14.463961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f48879-1a78-4d7c-8647-6ce0f6a78b60 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:14.541369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a27cf84-df2e-4f16-8d59-c0bff4d04c23 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:14.542032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a27cf84-df2e-4f16-8d59-c0bff4d04c23 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:14.542317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a27cf84-df2e-4f16-8d59-c0bff4d04c23 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:14.542591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a27cf84-df2e-4f16-8d59-c0bff4d04c23 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:14.544271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1a27cf84-df2e-4f16-8d59-c0bff4d04c23 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:14.637115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:14.653554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a27cf84-df2e-4f16-8d59-c0bff4d04c23 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:14.655025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a27cf84-df2e-4f16-8d59-c0bff4d04c23 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:14.674548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 660/1343] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:17.791705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d98278-6387-4f54-946a-7aa4b42da102 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:17.792234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d98278-6387-4f54-946a-7aa4b42da102 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:17.792234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d98278-6387-4f54-946a-7aa4b42da102 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:17.793681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 684/1344] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:17.799310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011a9ee-c2d5-4dcb-9fc8-4ced7154a676 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:17.799739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011a9ee-c2d5-4dcb-9fc8-4ced7154a676 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:17.799942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b011a9ee-c2d5-4dcb-9fc8-4ced7154a676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:17.864695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b011a9ee-c2d5-4dcb-9fc8-4ced7154a676 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:17.865922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b011a9ee-c2d5-4dcb-9fc8-4ced7154a676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:17.961507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 661/1345] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:17 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:18.000224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d98278-6387-4f54-946a-7aa4b42da102 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:18.001803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c9d98278-6387-4f54-946a-7aa4b42da102 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:18.090166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:18.107227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d98278-6387-4f54-946a-7aa4b42da102 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:18.182427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00320d0-91ce-4324-8cc6-93352a1dba63 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:18.183128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00320d0-91ce-4324-8cc6-93352a1dba63 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:18.183417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00320d0-91ce-4324-8cc6-93352a1dba63 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:18.183692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00320d0-91ce-4324-8cc6-93352a1dba63 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:18.185462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e00320d0-91ce-4324-8cc6-93352a1dba63 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:18.299043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00320d0-91ce-4324-8cc6-93352a1dba63 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:18.300489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00320d0-91ce-4324-8cc6-93352a1dba63 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:18.319397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 685/1346] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:17 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:22.561711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2bf408-4788-4cda-94c5-c447522f2a49 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:22.562292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2bf408-4788-4cda-94c5-c447522f2a49 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:22.562292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2bf408-4788-4cda-94c5-c447522f2a49 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:22.563767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 662/1347] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:22.569311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0634a413-ff5a-49c2-8506-26374776763d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:22.569792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0634a413-ff5a-49c2-8506-26374776763d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:22.569895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0634a413-ff5a-49c2-8506-26374776763d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:22.585251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:22.638837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0634a413-ff5a-49c2-8506-26374776763d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:22.640068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0634a413-ff5a-49c2-8506-26374776763d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:22.765098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 686/1348] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:22.804860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2bf408-4788-4cda-94c5-c447522f2a49 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:22.806986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-bb2bf408-4788-4cda-94c5-c447522f2a49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:22.877481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:22.927930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2bf408-4788-4cda-94c5-c447522f2a49 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:23.007208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca75e5ba-15d4-46ab-a3e4-af6e0d8b682d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:23.007792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca75e5ba-15d4-46ab-a3e4-af6e0d8b682d service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:23.008079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca75e5ba-15d4-46ab-a3e4-af6e0d8b682d service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:23.008353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca75e5ba-15d4-46ab-a3e4-af6e0d8b682d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:23.010185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ca75e5ba-15d4-46ab-a3e4-af6e0d8b682d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:23.123951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:23.127784 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca75e5ba-15d4-46ab-a3e4-af6e0d8b682d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:23.129247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca75e5ba-15d4-46ab-a3e4-af6e0d8b682d service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:23.147600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 663/1349] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:28.044987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39cc67-5873-41b3-bfc3-53a50107acf9 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:28.045444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39cc67-5873-41b3-bfc3-53a50107acf9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:28.045696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a39cc67-5873-41b3-bfc3-53a50107acf9 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:28.047461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 687/1350] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:10:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:28.053409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5ab6f-ea9f-4058-b128-bc48ed5870ca None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:28.053819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5ab6f-ea9f-4058-b128-bc48ed5870ca None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:28.054169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d5ab6f-ea9f-4058-b128-bc48ed5870ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:28.188131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-78d5ab6f-ea9f-4058-b128-bc48ed5870ca None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:28.190290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-78d5ab6f-ea9f-4058-b128-bc48ed5870ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:28.313963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 664/1351] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:10:28 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:32.442084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513131d2-560e-4831-826a-9746d6ff2c5f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:32.442600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513131d2-560e-4831-826a-9746d6ff2c5f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:32.442600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513131d2-560e-4831-826a-9746d6ff2c5f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:32.443962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 688/1352] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:10:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:32.450150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fc097-8137-4917-b264-28fc15048f51 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:32.450586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fc097-8137-4917-b264-28fc15048f51 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:32.450694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390fc097-8137-4917-b264-28fc15048f51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:32.507809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:32.532183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-390fc097-8137-4917-b264-28fc15048f51 None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:32.533911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-390fc097-8137-4917-b264-28fc15048f51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:32.644666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 665/1353] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:10:32 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:32.762798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=59 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:34.029237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d8e6e-389a-42a3-938c-eb9c03f2bafb None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:34.029849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d8e6e-389a-42a3-938c-eb9c03f2bafb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:34.029849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0d8e6e-389a-42a3-938c-eb9c03f2bafb None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:34.031124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 689/1354] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:10:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:34.036629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ea4c02-3b5d-4ded-9ace-ffeba7fb326e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:34.037077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ea4c02-3b5d-4ded-9ace-ffeba7fb326e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:34.037353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3ea4c02-3b5d-4ded-9ace-ffeba7fb326e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:34.113417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f3ea4c02-3b5d-4ded-9ace-ffeba7fb326e None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:34.114637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f3ea4c02-3b5d-4ded-9ace-ffeba7fb326e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:34.158716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:34.224575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 666/1355] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:10:34 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:10:37.693736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ed6c24-229d-45c4-b217-340613e31815 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:37.694606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ed6c24-229d-45c4-b217-340613e31815 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:37.695088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ed6c24-229d-45c4-b217-340613e31815 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:37.696897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 690/1356] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:37.706806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60e12c1-2baf-4a51-9241-5d7026ff03fa None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:37.707331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60e12c1-2baf-4a51-9241-5d7026ff03fa None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:37.708609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60e12c1-2baf-4a51-9241-5d7026ff03fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:37.796796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e60e12c1-2baf-4a51-9241-5d7026ff03fa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:37.798579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e60e12c1-2baf-4a51-9241-5d7026ff03fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:37.908787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 667/1357] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:37.964366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1461cc3-87a2-4594-ba81-457ad3cd9c1a None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:37.964654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1461cc3-87a2-4594-ba81-457ad3cd9c1a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:37.965144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1461cc3-87a2-4594-ba81-457ad3cd9c1a None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:37.966436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 691/1358] 10.222.0.27 () {54 vars in 961 bytes} [Thu Jul 14 12:10:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:37.974758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63be92f-823a-427d-9278-5c68ab19804a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:37.975078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63be92f-823a-427d-9278-5c68ab19804a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:37.975358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63be92f-823a-427d-9278-5c68ab19804a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:38.000113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:38.064938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c63be92f-823a-427d-9278-5c68ab19804a None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:38.066134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c63be92f-823a-427d-9278-5c68ab19804a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:38.201775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 668/1359] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Jul 14 12:10:37 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:38.202603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1461cc3-87a2-4594-ba81-457ad3cd9c1a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:38.204258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c1461cc3-87a2-4594-ba81-457ad3cd9c1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:38.378243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1461cc3-87a2-4594-ba81-457ad3cd9c1a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:38.456216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:38.497691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da36c696-002c-4916-a27d-f8d0eaff847a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:38.498347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36c696-002c-4916-a27d-f8d0eaff847a service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:38.498628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36c696-002c-4916-a27d-f8d0eaff847a service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:38.498942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da36c696-002c-4916-a27d-f8d0eaff847a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:38.500607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-da36c696-002c-4916-a27d-f8d0eaff847a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:38.601573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da36c696-002c-4916-a27d-f8d0eaff847a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:38.603389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da36c696-002c-4916-a27d-f8d0eaff847a service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:38.622420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 692/1360] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 420 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:40.879136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a3d6d-6921-47de-a018-07b12e779aac None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:40.879638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a3d6d-6921-47de-a018-07b12e779aac None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:40.879638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879a3d6d-6921-47de-a018-07b12e779aac None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:40.881129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 669/1361] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:40.885972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2047686b-910a-4141-ada9-ccdc6cad6c60 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:40.886367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2047686b-910a-4141-ada9-ccdc6cad6c60 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:40.886563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2047686b-910a-4141-ada9-ccdc6cad6c60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:40.958234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2047686b-910a-4141-ada9-ccdc6cad6c60 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:40.959398 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2047686b-910a-4141-ada9-ccdc6cad6c60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:40.998181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:41.061491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 693/1362] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:41.105790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a3d6d-6921-47de-a018-07b12e779aac None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:41.107810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-879a3d6d-6921-47de-a018-07b12e779aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:41.185997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:41.290315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879a3d6d-6921-47de-a018-07b12e779aac None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:41.415575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1eebfe-55c1-4494-95c2-7715b1fa8854 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:41.416343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1eebfe-55c1-4494-95c2-7715b1fa8854 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:41.416710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1eebfe-55c1-4494-95c2-7715b1fa8854 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:41.417061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1eebfe-55c1-4494-95c2-7715b1fa8854 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:41.418861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1e1eebfe-55c1-4494-95c2-7715b1fa8854 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:41.466201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:41.544462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1eebfe-55c1-4494-95c2-7715b1fa8854 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:41.546094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1eebfe-55c1-4494-95c2-7715b1fa8854 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:41.569317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 670/1363] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 464 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:44.120575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812feb9b-527d-4e0f-a06e-56ee7dd063eb None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:44.121082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812feb9b-527d-4e0f-a06e-56ee7dd063eb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:44.121082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812feb9b-527d-4e0f-a06e-56ee7dd063eb None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:44.122652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 694/1364] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:44.128565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102007a-31d4-4581-984c-bd0c97ffd74f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:44.129061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102007a-31d4-4581-984c-bd0c97ffd74f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:44.129157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c102007a-31d4-4581-984c-bd0c97ffd74f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:44.201104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c102007a-31d4-4581-984c-bd0c97ffd74f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:44.202334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c102007a-31d4-4581-984c-bd0c97ffd74f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:44.305225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:44.308133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 671/1365] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:44.355257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc44810-f34c-45f7-a979-150d7edda48f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:44.355545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc44810-f34c-45f7-a979-150d7edda48f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:44.355825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cc44810-f34c-45f7-a979-150d7edda48f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:44.357243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 695/1366] 10.222.0.27 () {54 vars in 961 bytes} [Thu Jul 14 12:10:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:44.363504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f5984b-6929-4542-b7d6-c026b63bddc6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:44.365680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f5984b-6929-4542-b7d6-c026b63bddc6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:44.365680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f5984b-6929-4542-b7d6-c026b63bddc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:44.455428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b6f5984b-6929-4542-b7d6-c026b63bddc6 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:44.462201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b6f5984b-6929-4542-b7d6-c026b63bddc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:44.591282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 672/1367] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Jul 14 12:10:44 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 229 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:44.599958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc44810-f34c-45f7-a979-150d7edda48f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:44.601897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9cc44810-f34c-45f7-a979-150d7edda48f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:44.770552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cc44810-f34c-45f7-a979-150d7edda48f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:44.792311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:44.891651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841f1d5c-9638-442a-8f69-9b261723d234 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:44.892303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841f1d5c-9638-442a-8f69-9b261723d234 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:44.892714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841f1d5c-9638-442a-8f69-9b261723d234 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:44.893000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841f1d5c-9638-442a-8f69-9b261723d234 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:44.894639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-841f1d5c-9638-442a-8f69-9b261723d234 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:44.992990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841f1d5c-9638-442a-8f69-9b261723d234 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:44.994503 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841f1d5c-9638-442a-8f69-9b261723d234 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:45.018823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 696/1368] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 417 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:45.996419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1887da2-0b4a-4774-b7c5-3210acdfcedf None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:45.996928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1887da2-0b4a-4774-b7c5-3210acdfcedf None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:45.996928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1887da2-0b4a-4774-b7c5-3210acdfcedf None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:46.002700 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 673/1369] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:46.008563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2be607-40c5-45bd-8626-912984fc7999 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:46.009018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2be607-40c5-45bd-8626-912984fc7999 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:46.009120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db2be607-40c5-45bd-8626-912984fc7999 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:46.045835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:46.093442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-db2be607-40c5-45bd-8626-912984fc7999 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:46.096109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-db2be607-40c5-45bd-8626-912984fc7999 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:46.215539 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 697/1370] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:10:46.565551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7219a099-5dc7-45f7-b2fe-8e965d6df484 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:46.565551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7219a099-5dc7-45f7-b2fe-8e965d6df484 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:46.565551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7219a099-5dc7-45f7-b2fe-8e965d6df484 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:46.568164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 674/1371] 10.222.0.27 () {54 vars in 963 bytes} [Thu Jul 14 12:10:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:46.571519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623c59a-6839-43f7-8fcd-1a69d407c426 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:46.571812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623c59a-6839-43f7-8fcd-1a69d407c426 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:46.572092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f623c59a-6839-43f7-8fcd-1a69d407c426 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:46.651505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f623c59a-6839-43f7-8fcd-1a69d407c426 None None] MFA Rules not processed for user `e114b292a7f544b0b24016199edcef25`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:46.652723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f623c59a-6839-43f7-8fcd-1a69d407c426 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:46.707528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:46.744621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 698/1372] 10.222.0.27 () {58 vars in 1060 bytes} [Thu Jul 14 12:10:46 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:46.753117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7219a099-5dc7-45f7-b2fe-8e965d6df484 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:46.754638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7219a099-5dc7-45f7-b2fe-8e965d6df484 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:46.806192 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:46.865216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7219a099-5dc7-45f7-b2fe-8e965d6df484 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:46.944270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05067f25-a2f2-413b-87a5-7b82df9ee4d9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:46.944927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05067f25-a2f2-413b-87a5-7b82df9ee4d9 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:46.945211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05067f25-a2f2-413b-87a5-7b82df9ee4d9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:46.945488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05067f25-a2f2-413b-87a5-7b82df9ee4d9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:46.947205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-05067f25-a2f2-413b-87a5-7b82df9ee4d9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:47.050742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:47.065346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05067f25-a2f2-413b-87a5-7b82df9ee4d9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:47.066066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05067f25-a2f2-413b-87a5-7b82df9ee4d9 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:47.096457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 675/1373] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:47.251873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa1194-5a2a-4433-ae3c-bafd77cb188b None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:47.252881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa1194-5a2a-4433-ae3c-bafd77cb188b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:47.253219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa1194-5a2a-4433-ae3c-bafd77cb188b None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:47.254852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 699/1374] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:47.259789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fffb9ac-cd94-4ad4-8716-17dc34f559c0 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:47.259978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fffb9ac-cd94-4ad4-8716-17dc34f559c0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:47.260284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fffb9ac-cd94-4ad4-8716-17dc34f559c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:47.331344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9fffb9ac-cd94-4ad4-8716-17dc34f559c0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:47.332715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9fffb9ac-cd94-4ad4-8716-17dc34f559c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:47.447355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 676/1375] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:47.499583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa1194-5a2a-4433-ae3c-bafd77cb188b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:47.501718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e2fa1194-5a2a-4433-ae3c-bafd77cb188b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:47.666037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa1194-5a2a-4433-ae3c-bafd77cb188b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:47.720713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:47.786536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f444d2d-b9e1-4218-94ee-a899aa44e952 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:47.787223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f444d2d-b9e1-4218-94ee-a899aa44e952 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:47.787510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f444d2d-b9e1-4218-94ee-a899aa44e952 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:47.787787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f444d2d-b9e1-4218-94ee-a899aa44e952 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:47.789476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6f444d2d-b9e1-4218-94ee-a899aa44e952 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:47.879133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f444d2d-b9e1-4218-94ee-a899aa44e952 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:47.880749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f444d2d-b9e1-4218-94ee-a899aa44e952 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:47.899710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 700/1376] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:50.063333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0068074-065c-471e-a191-7c7585c57855 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:50.063874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0068074-065c-471e-a191-7c7585c57855 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:50.063874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0068074-065c-471e-a191-7c7585c57855 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:50.065304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 677/1377] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:50.070651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f945c6-143b-4534-b9bf-fbe11de24682 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:50.071179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f945c6-143b-4534-b9bf-fbe11de24682 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:50.071291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f945c6-143b-4534-b9bf-fbe11de24682 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:50.154631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e7f945c6-143b-4534-b9bf-fbe11de24682 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:50.154631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e7f945c6-143b-4534-b9bf-fbe11de24682 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:50.162800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:50.283652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 701/1378] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:50.344611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0068074-065c-471e-a191-7c7585c57855 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:50.344611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e0068074-065c-471e-a191-7c7585c57855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:50.392711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:50.476885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0068074-065c-471e-a191-7c7585c57855 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:50.596173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a10b7f52-0468-487a-8fa8-f35c6349fde4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:50.596576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b7f52-0468-487a-8fa8-f35c6349fde4 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:50.596860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b7f52-0468-487a-8fa8-f35c6349fde4 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:50.597137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a10b7f52-0468-487a-8fa8-f35c6349fde4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:50.598816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a10b7f52-0468-487a-8fa8-f35c6349fde4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:50.623526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:50.713659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10b7f52-0468-487a-8fa8-f35c6349fde4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:50.715672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a10b7f52-0468-487a-8fa8-f35c6349fde4 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:50.737473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 678/1379] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:52.953868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5332f1-fe6e-4620-8e19-593fcd3cd32f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:52.954570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5332f1-fe6e-4620-8e19-593fcd3cd32f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:52.954674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5332f1-fe6e-4620-8e19-593fcd3cd32f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:52.956295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 702/1380] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:52.961339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e82cab-85fb-4108-b1c3-2cb2f58b54c6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:52.961798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e82cab-85fb-4108-b1c3-2cb2f58b54c6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:52.961891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e82cab-85fb-4108-b1c3-2cb2f58b54c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:53.029754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-28e82cab-85fb-4108-b1c3-2cb2f58b54c6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:53.030976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-28e82cab-85fb-4108-b1c3-2cb2f58b54c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:53.139848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:53.148410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 679/1381] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:53.193060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5332f1-fe6e-4620-8e19-593fcd3cd32f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:53.194551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8f5332f1-fe6e-4620-8e19-593fcd3cd32f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:53.331363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:53.342399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5332f1-fe6e-4620-8e19-593fcd3cd32f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:53.465526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f566bd45-c38c-4960-850a-e826c3331082 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:53.466177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f566bd45-c38c-4960-850a-e826c3331082 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:53.466461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f566bd45-c38c-4960-850a-e826c3331082 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:53.466734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f566bd45-c38c-4960-850a-e826c3331082 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:53.468549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f566bd45-c38c-4960-850a-e826c3331082 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:53.563815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f566bd45-c38c-4960-850a-e826c3331082 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:53.565362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f566bd45-c38c-4960-850a-e826c3331082 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:53.577657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:53.586316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 703/1382] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:55.840201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9343758-f797-417d-a9b4-2731491ce7b9 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:55.840770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9343758-f797-417d-a9b4-2731491ce7b9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:55.840770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9343758-f797-417d-a9b4-2731491ce7b9 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:55.842267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 680/1383] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:55.852018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979bf241-d0e0-49e0-a3fb-7747ceff0669 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:55.852636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979bf241-d0e0-49e0-a3fb-7747ceff0669 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:55.852636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-979bf241-d0e0-49e0-a3fb-7747ceff0669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:55.927165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-979bf241-d0e0-49e0-a3fb-7747ceff0669 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:55.928764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-979bf241-d0e0-49e0-a3fb-7747ceff0669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:56.035207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 704/1384] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:56.078193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9343758-f797-417d-a9b4-2731491ce7b9 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:56.080501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f9343758-f797-417d-a9b4-2731491ce7b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:56.255335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9343758-f797-417d-a9b4-2731491ce7b9 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:56.341597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:56.388026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fbb7dab-bdbe-4efa-8f47-6a0e13a96b5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:56.388695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbb7dab-bdbe-4efa-8f47-6a0e13a96b5c service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:56.388985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbb7dab-bdbe-4efa-8f47-6a0e13a96b5c service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:56.389268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fbb7dab-bdbe-4efa-8f47-6a0e13a96b5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:56.391124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9fbb7dab-bdbe-4efa-8f47-6a0e13a96b5c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:56.504221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbb7dab-bdbe-4efa-8f47-6a0e13a96b5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:56.505880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fbb7dab-bdbe-4efa-8f47-6a0e13a96b5c service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:56.528005 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 681/1385] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:10:58.856900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d810c7a-ced9-4853-a1b5-250a80808646 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:58.857460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d810c7a-ced9-4853-a1b5-250a80808646 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:58.857460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d810c7a-ced9-4853-a1b5-250a80808646 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:58.858879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 705/1386] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:10:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:10:58.864805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552604db-2068-41e8-8985-d20a3a2f43fe None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:58.865244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552604db-2068-41e8-8985-d20a3a2f43fe None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:58.865602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552604db-2068-41e8-8985-d20a3a2f43fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:58.932178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-552604db-2068-41e8-8985-d20a3a2f43fe None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:10:58.933307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-552604db-2068-41e8-8985-d20a3a2f43fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:58.967200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:59.036460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 682/1387] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:10:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:10:59.086157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d810c7a-ced9-4853-a1b5-250a80808646 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:10:59.088127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6d810c7a-ced9-4853-a1b5-250a80808646 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:59.172764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:59.248452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d810c7a-ced9-4853-a1b5-250a80808646 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:10:59.369081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b661c62b-0d95-4629-aba3-f0107125f045 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:10:59.369822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b661c62b-0d95-4629-aba3-f0107125f045 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:10:59.370141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b661c62b-0d95-4629-aba3-f0107125f045 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:10:59.370454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b661c62b-0d95-4629-aba3-f0107125f045 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:10:59.372376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b661c62b-0d95-4629-aba3-f0107125f045 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:10:59.414250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:10:59.472112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b661c62b-0d95-4629-aba3-f0107125f045 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:10:59.473908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b661c62b-0d95-4629-aba3-f0107125f045 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:10:59.494137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 706/1388] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:10:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:01.788000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982ca20f-7b5f-43b1-866b-19e0d2707b4f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:01.788603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982ca20f-7b5f-43b1-866b-19e0d2707b4f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:01.788603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982ca20f-7b5f-43b1-866b-19e0d2707b4f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:01.790017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 683/1389] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:11:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:01.795241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d0aca-66ad-432d-803c-f51650624643 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:01.795668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d0aca-66ad-432d-803c-f51650624643 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:01.795776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61d0aca-66ad-432d-803c-f51650624643 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:01.860627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b61d0aca-66ad-432d-803c-f51650624643 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:01.861857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b61d0aca-66ad-432d-803c-f51650624643 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:01.959334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 707/1390] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:11:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:02.003694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982ca20f-7b5f-43b1-866b-19e0d2707b4f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:02.005376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-982ca20f-7b5f-43b1-866b-19e0d2707b4f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:02.153993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982ca20f-7b5f-43b1-866b-19e0d2707b4f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:02.170852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:02.270162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-274dec69-bb4d-4bc1-a634-c70f5d77b23b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:02.270730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274dec69-bb4d-4bc1-a634-c70f5d77b23b service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:02.271125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274dec69-bb4d-4bc1-a634-c70f5d77b23b service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:02.271407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-274dec69-bb4d-4bc1-a634-c70f5d77b23b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:02.273069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-274dec69-bb4d-4bc1-a634-c70f5d77b23b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:02.361802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274dec69-bb4d-4bc1-a634-c70f5d77b23b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:02.363552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-274dec69-bb4d-4bc1-a634-c70f5d77b23b service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:02.381359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 684/1391] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:02.662803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085516b-a8d1-421d-9f08-203c60b91ad5 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:02.663090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085516b-a8d1-421d-9f08-203c60b91ad5 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:02.663383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f085516b-a8d1-421d-9f08-203c60b91ad5 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:02.664768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 708/1392] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:11:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:02.670034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e19b02-40c9-462c-82f1-1ace54d91483 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:02.670326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e19b02-40c9-462c-82f1-1ace54d91483 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:02.670635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17e19b02-40c9-462c-82f1-1ace54d91483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:02.697573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:02.753742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-17e19b02-40c9-462c-82f1-1ace54d91483 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:02.755094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-17e19b02-40c9-462c-82f1-1ace54d91483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:02.873964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 685/1393] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:11:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:02.932438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab5044-cc8a-4a35-97a5-b94d508e570f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:02.932992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab5044-cc8a-4a35-97a5-b94d508e570f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:02.932992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ab5044-cc8a-4a35-97a5-b94d508e570f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:02.934511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 709/1394] 10.222.0.27 () {54 vars in 962 bytes} [Thu Jul 14 12:11:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:02.940144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850d8319-bb09-4667-be5f-4fcd75b03dc4 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:02.940511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850d8319-bb09-4667-be5f-4fcd75b03dc4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:02.943477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-850d8319-bb09-4667-be5f-4fcd75b03dc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:03.021906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-850d8319-bb09-4667-be5f-4fcd75b03dc4 None None] MFA Rules not processed for user `86ace7e877e3484b8853536b77c76885`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:03.023151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-850d8319-bb09-4667-be5f-4fcd75b03dc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:03.067009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:03.103791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 686/1395] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Jul 14 12:11:02 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:03.111840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab5044-cc8a-4a35-97a5-b94d508e570f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:03.113358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-32ab5044-cc8a-4a35-97a5-b94d508e570f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:03.127763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:03.221183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ab5044-cc8a-4a35-97a5-b94d508e570f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:03.293914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88236d48-2263-45c0-aa1b-f4dc3b7f6cab service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:03.294582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88236d48-2263-45c0-aa1b-f4dc3b7f6cab service glance] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:03.294926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88236d48-2263-45c0-aa1b-f4dc3b7f6cab service glance] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:03.295243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88236d48-2263-45c0-aa1b-f4dc3b7f6cab service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:03.297003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-88236d48-2263-45c0-aa1b-f4dc3b7f6cab service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:03.366700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:03.408893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88236d48-2263-45c0-aa1b-f4dc3b7f6cab service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:03.410316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88236d48-2263-45c0-aa1b-f4dc3b7f6cab service glance] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:03.430310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 710/1396] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:11:04.291275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b86f7c-09ea-487d-98a9-cd7df3fabd5b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:04.291768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b86f7c-09ea-487d-98a9-cd7df3fabd5b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:04.291883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b86f7c-09ea-487d-98a9-cd7df3fabd5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:04.357780 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-41b86f7c-09ea-487d-98a9-cd7df3fabd5b None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:04.359023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-41b86f7c-09ea-487d-98a9-cd7df3fabd5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:04.446878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 687/1397] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:11:04 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 157 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:04.491752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b3b8a-f2f6-494c-8f0a-84f0ac851929 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:04.492428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b3b8a-f2f6-494c-8f0a-84f0ac851929 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:04.492428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b4b3b8a-f2f6-494c-8f0a-84f0ac851929 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:04.493708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 711/1398] 10.222.0.27 () {54 vars in 942 bytes} [Thu Jul 14 12:11:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:04.501791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a79cc8c-1374-47e7-b251-3d5d5c37f404 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:04.502086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a79cc8c-1374-47e7-b251-3d5d5c37f404 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:04.502371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a79cc8c-1374-47e7-b251-3d5d5c37f404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:04.571475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:04.580299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7a79cc8c-1374-47e7-b251-3d5d5c37f404 None None] MFA Rules not processed for user `7b1634a6e6ec4db9a8851b1329ffe6a0`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:04.581506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7a79cc8c-1374-47e7-b251-3d5d5c37f404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:04.688849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 688/1399] 10.222.0.27 () {58 vars in 1039 bytes} [Thu Jul 14 12:11:04 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:11:04.698793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b3b8a-f2f6-494c-8f0a-84f0ac851929 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:04.700300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4b4b3b8a-f2f6-494c-8f0a-84f0ac851929 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:04.721248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eab561-b38b-41ed-9d75-cbf40df0d0eb None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:04.721557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eab561-b38b-41ed-9d75-cbf40df0d0eb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:04.721826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4eab561-b38b-41ed-9d75-cbf40df0d0eb None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:04.723256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 689/1400] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:11:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:04.728397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:04.728689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:04.728971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:04.793404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:04.794618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:04.850859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b4b3b8a-f2f6-494c-8f0a-84f0ac851929 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:04.858934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:04.888352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:04.895964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 690/1401] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:11:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:11:04.941547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:04.942811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:04.957451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc7ecb2-c794-4d2e-9d1a-5c83277ed6c1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:04.958104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc7ecb2-c794-4d2e-9d1a-5c83277ed6c1 service swift] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:04.958386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc7ecb2-c794-4d2e-9d1a-5c83277ed6c1 service swift] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:04.958660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc7ecb2-c794-4d2e-9d1a-5c83277ed6c1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:04.960367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5bc7ecb2-c794-4d2e-9d1a-5c83277ed6c1 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:05.043171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc7ecb2-c794-4d2e-9d1a-5c83277ed6c1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:05.044786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc7ecb2-c794-4d2e-9d1a-5c83277ed6c1 service swift] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:05.053017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 712/1402] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:11:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:11:05.059531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c4ac7e1-e7ea-4954-b8cf-7cdbd0380726 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:05.134811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:05.175480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e1c1a4-db35-4a25-b73d-16afd318e65d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:05.176128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1c1a4-db35-4a25-b73d-16afd318e65d service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:05.176413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1c1a4-db35-4a25-b73d-16afd318e65d service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:05.176689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1c1a4-db35-4a25-b73d-16afd318e65d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:05.178335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e2e1c1a4-db35-4a25-b73d-16afd318e65d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:05.268058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e1c1a4-db35-4a25-b73d-16afd318e65d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:05.269674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e1c1a4-db35-4a25-b73d-16afd318e65d service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:05.288644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 691/1403] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:07.453682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8f0a8-e26a-4bf6-a7ed-4e5b9fa15d13 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:07.454306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8f0a8-e26a-4bf6-a7ed-4e5b9fa15d13 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:07.454306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f8f0a8-e26a-4bf6-a7ed-4e5b9fa15d13 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:07.457634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 713/1404] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:11:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:07.460755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b87d7a-bdd8-417e-9a94-4fbc089d7b5d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:07.461188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b87d7a-bdd8-417e-9a94-4fbc089d7b5d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:07.461296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b87d7a-bdd8-417e-9a94-4fbc089d7b5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:07.532422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d0b87d7a-bdd8-417e-9a94-4fbc089d7b5d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:07.533650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d0b87d7a-bdd8-417e-9a94-4fbc089d7b5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:07.575083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:07.644699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 692/1405] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:11:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:07.689575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8f0a8-e26a-4bf6-a7ed-4e5b9fa15d13 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:07.691158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-29f8f0a8-e26a-4bf6-a7ed-4e5b9fa15d13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:07.779186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:07.858099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f8f0a8-e26a-4bf6-a7ed-4e5b9fa15d13 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:07.990669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-113b79dc-eac4-4b92-8c3d-63e4a0b67960 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:07.991752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113b79dc-eac4-4b92-8c3d-63e4a0b67960 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:07.992041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113b79dc-eac4-4b92-8c3d-63e4a0b67960 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:07.992321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113b79dc-eac4-4b92-8c3d-63e4a0b67960 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:07.993981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-113b79dc-eac4-4b92-8c3d-63e4a0b67960 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:08.027266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:08.097110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113b79dc-eac4-4b92-8c3d-63e4a0b67960 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:08.099166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-113b79dc-eac4-4b92-8c3d-63e4a0b67960 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:08.118448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 714/1406] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:10.420597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425f40d4-4181-4471-9389-f44d859ba1bc None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:10.421842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425f40d4-4181-4471-9389-f44d859ba1bc None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:10.422394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-425f40d4-4181-4471-9389-f44d859ba1bc None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:10.424335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 693/1407] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:11:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:10.432869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd779b6-282e-4f5f-80e8-abc3edb20c63 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:10.433443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd779b6-282e-4f5f-80e8-abc3edb20c63 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:10.433443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd779b6-282e-4f5f-80e8-abc3edb20c63 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:10.507894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ddd779b6-282e-4f5f-80e8-abc3edb20c63 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:10.509035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ddd779b6-282e-4f5f-80e8-abc3edb20c63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:10.618717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 715/1408] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:11:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:10.664674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425f40d4-4181-4471-9389-f44d859ba1bc None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:10.666180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-425f40d4-4181-4471-9389-f44d859ba1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:10.836261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-425f40d4-4181-4471-9389-f44d859ba1bc None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:10.863611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:10.961673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-054aa84a-7afb-46c6-9cce-09e4974f21df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:10.962321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054aa84a-7afb-46c6-9cce-09e4974f21df service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:10.962609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054aa84a-7afb-46c6-9cce-09e4974f21df service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:10.962921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-054aa84a-7afb-46c6-9cce-09e4974f21df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:10.968355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-054aa84a-7afb-46c6-9cce-09e4974f21df service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:11.065080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054aa84a-7afb-46c6-9cce-09e4974f21df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:11.066835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-054aa84a-7afb-46c6-9cce-09e4974f21df service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:11.095311 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 694/1409] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:20.616855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:21.092443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=88 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:28.405090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ace57bb-fe12-428c-aedf-83921101367f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:28.406310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ace57bb-fe12-428c-aedf-83921101367f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:28.406310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ace57bb-fe12-428c-aedf-83921101367f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:28.408007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 716/1410] 10.222.0.27 () {54 vars in 918 bytes} [Thu Jul 14 12:11:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:28.413239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dcb6a9-7936-494a-a470-1a714ce8cd52 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:28.413635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dcb6a9-7936-494a-a470-1a714ce8cd52 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:28.413792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8dcb6a9-7936-494a-a470-1a714ce8cd52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:28.488045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b8dcb6a9-7936-494a-a470-1a714ce8cd52 None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:28.489275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b8dcb6a9-7936-494a-a470-1a714ce8cd52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:28.594410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 695/1411] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Jul 14 12:11:28 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:28.609411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1798f15-fab7-43da-b654-b23d7c2ee523 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:28.609701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1798f15-fab7-43da-b654-b23d7c2ee523 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:28.610039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1798f15-fab7-43da-b654-b23d7c2ee523 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:28.611460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 717/1412] 10.222.0.27 () {54 vars in 918 bytes} [Thu Jul 14 12:11:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:28.616400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b65a0-4c10-4ec9-923b-6584b98dd9d2 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:28.616690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b65a0-4c10-4ec9-923b-6584b98dd9d2 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:28.616969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-426b65a0-4c10-4ec9-923b-6584b98dd9d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:28.688539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:28.691664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-426b65a0-4c10-4ec9-923b-6584b98dd9d2 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:28.693624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-426b65a0-4c10-4ec9-923b-6584b98dd9d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:28.813901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 696/1413] 10.222.0.27 () {58 vars in 1015 bytes} [Thu Jul 14 12:11:28 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:31.307607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c66134-b724-4e56-be2e-4a6187156b68 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:31.308185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c66134-b724-4e56-be2e-4a6187156b68 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:31.308185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c66134-b724-4e56-be2e-4a6187156b68 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:31.309546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 718/1414] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:11:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:31.314999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db706e-a052-430e-82f2-88a32d7bdb9c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:31.315431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db706e-a052-430e-82f2-88a32d7bdb9c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:31.315561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1db706e-a052-430e-82f2-88a32d7bdb9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:31.381112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b1db706e-a052-430e-82f2-88a32d7bdb9c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:31.382327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b1db706e-a052-430e-82f2-88a32d7bdb9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:31.438693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:31.481691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 697/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:11:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:31.525700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c66134-b724-4e56-be2e-4a6187156b68 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:31.527269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-03c66134-b724-4e56-be2e-4a6187156b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:31.672269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c66134-b724-4e56-be2e-4a6187156b68 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:31.744601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:31.785824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8cd3c69-88fd-4bdb-b5b3-5688e4ca783b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:31.786471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cd3c69-88fd-4bdb-b5b3-5688e4ca783b service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:31.786758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cd3c69-88fd-4bdb-b5b3-5688e4ca783b service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:31.787067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8cd3c69-88fd-4bdb-b5b3-5688e4ca783b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:31.788712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c8cd3c69-88fd-4bdb-b5b3-5688e4ca783b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:31.876439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cd3c69-88fd-4bdb-b5b3-5688e4ca783b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:31.878037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8cd3c69-88fd-4bdb-b5b3-5688e4ca783b service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:31.896133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 719/1416] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:32.798959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4b43-8595-4035-95c5-7d3954d7a771 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:32.799503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4b43-8595-4035-95c5-7d3954d7a771 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:32.799503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df4b43-8595-4035-95c5-7d3954d7a771 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:32.800896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 698/1417] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:32.805838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08a29d3-d053-4d3b-822d-52d676d29492 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:32.806270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08a29d3-d053-4d3b-822d-52d676d29492 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:32.806405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a08a29d3-d053-4d3b-822d-52d676d29492 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:32.880573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a08a29d3-d053-4d3b-822d-52d676d29492 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:32.881784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a08a29d3-d053-4d3b-822d-52d676d29492 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:32.909634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:32.979010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 720/1418] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:33.129212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df4b43-8595-4035-95c5-7d3954d7a771 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:33.130733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-96df4b43-8595-4035-95c5-7d3954d7a771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:33.276490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df4b43-8595-4035-95c5-7d3954d7a771 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:33.316906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:33.389659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fd8f22-c18b-4c8b-9562-3ed86df84758 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:33.390557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fd8f22-c18b-4c8b-9562-3ed86df84758 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:33.390874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fd8f22-c18b-4c8b-9562-3ed86df84758 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:33.391155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fd8f22-c18b-4c8b-9562-3ed86df84758 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:33.392835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-72fd8f22-c18b-4c8b-9562-3ed86df84758 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:33.481535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fd8f22-c18b-4c8b-9562-3ed86df84758 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:33.483261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fd8f22-c18b-4c8b-9562-3ed86df84758 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:33.501129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 699/1419] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:34.386056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03ea42e-1773-4ada-bcf6-ed90e25fe1a1 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:34.386679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03ea42e-1773-4ada-bcf6-ed90e25fe1a1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:34.386679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03ea42e-1773-4ada-bcf6-ed90e25fe1a1 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:34.388052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 721/1420] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:34.393708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e4a4c-afc6-4fba-b7aa-ec5df920e164 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:34.394153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e4a4c-afc6-4fba-b7aa-ec5df920e164 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:34.394260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e4a4c-afc6-4fba-b7aa-ec5df920e164 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:34.447605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:34.471378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-471e4a4c-afc6-4fba-b7aa-ec5df920e164 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:34.472667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-471e4a4c-afc6-4fba-b7aa-ec5df920e164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:34.569758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 700/1421] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:34.719659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03ea42e-1773-4ada-bcf6-ed90e25fe1a1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:34.721184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d03ea42e-1773-4ada-bcf6-ed90e25fe1a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:34.866934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d03ea42e-1773-4ada-bcf6-ed90e25fe1a1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:34.879391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:34.979279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-001d8284-28bb-4a48-bae5-e8b04a47687b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:34.979845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d8284-28bb-4a48-bae5-e8b04a47687b service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:34.980131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d8284-28bb-4a48-bae5-e8b04a47687b service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:34.980404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-001d8284-28bb-4a48-bae5-e8b04a47687b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:34.982037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-001d8284-28bb-4a48-bae5-e8b04a47687b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:35.069801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001d8284-28bb-4a48-bae5-e8b04a47687b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:35.071426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-001d8284-28bb-4a48-bae5-e8b04a47687b service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:35.089015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 722/1422] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:11:35.962026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a824b-680b-4092-818a-e2a260a7118e None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:35.962567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a824b-680b-4092-818a-e2a260a7118e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:35.962567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3a824b-680b-4092-818a-e2a260a7118e None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:35.964020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 701/1423] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:35.968937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f63051-f1a8-4c9e-9307-2e475da55ce8 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:35.969350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f63051-f1a8-4c9e-9307-2e475da55ce8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:35.969530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f63051-f1a8-4c9e-9307-2e475da55ce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:35.988605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:36.044748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d0f63051-f1a8-4c9e-9307-2e475da55ce8 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:36.046165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d0f63051-f1a8-4c9e-9307-2e475da55ce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:36.143273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 723/1424] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:36.293380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3a824b-680b-4092-818a-e2a260a7118e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:36.294866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8b3a824b-680b-4092-818a-e2a260a7118e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:36.403609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:36.441711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3a824b-680b-4092-818a-e2a260a7118e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:36.553064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323f39a5-5535-44cb-9248-e7fb63488dc7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:36.553622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323f39a5-5535-44cb-9248-e7fb63488dc7 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:36.553910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323f39a5-5535-44cb-9248-e7fb63488dc7 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:36.554190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323f39a5-5535-44cb-9248-e7fb63488dc7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:36.555872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-323f39a5-5535-44cb-9248-e7fb63488dc7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:36.631078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:36.644254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323f39a5-5535-44cb-9248-e7fb63488dc7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:36.645865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323f39a5-5535-44cb-9248-e7fb63488dc7 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:36.664637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 702/1425] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:37.532199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b864ba-313d-4a58-9345-5b90448cfb6f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:37.533136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b864ba-313d-4a58-9345-5b90448cfb6f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:37.533136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b864ba-313d-4a58-9345-5b90448cfb6f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:37.534421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 724/1426] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:37.539381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae70d87-327b-4b9b-9171-4e281c1a8b73 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:37.539821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae70d87-327b-4b9b-9171-4e281c1a8b73 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:37.539942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ae70d87-327b-4b9b-9171-4e281c1a8b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:37.613274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4ae70d87-327b-4b9b-9171-4e281c1a8b73 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:37.614461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4ae70d87-327b-4b9b-9171-4e281c1a8b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:37.711046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 703/1427] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:37.862303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b864ba-313d-4a58-9345-5b90448cfb6f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:37.863753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b7b864ba-313d-4a58-9345-5b90448cfb6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:37.948851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:38.010519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b864ba-313d-4a58-9345-5b90448cfb6f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:38.121384 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efd68c3-41c3-4cb4-85b2-c51a3fb4c2b0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:38.121957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd68c3-41c3-4cb4-85b2-c51a3fb4c2b0 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:38.122238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd68c3-41c3-4cb4-85b2-c51a3fb4c2b0 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:38.122522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efd68c3-41c3-4cb4-85b2-c51a3fb4c2b0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:38.124814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8efd68c3-41c3-4cb4-85b2-c51a3fb4c2b0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:38.171696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:38.213645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd68c3-41c3-4cb4-85b2-c51a3fb4c2b0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:38.215284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efd68c3-41c3-4cb4-85b2-c51a3fb4c2b0 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:38.233670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 725/1428] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:39.106496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c3a35-bb69-481e-b12f-d23658117c52 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:39.107073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c3a35-bb69-481e-b12f-d23658117c52 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:39.107073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39c3a35-bb69-481e-b12f-d23658117c52 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:39.109141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 704/1429] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:39.114117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6553701-dc86-4d45-a82f-f3f0ae40303e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:39.114522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6553701-dc86-4d45-a82f-f3f0ae40303e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:39.114685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6553701-dc86-4d45-a82f-f3f0ae40303e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:39.188826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c6553701-dc86-4d45-a82f-f3f0ae40303e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:39.190034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c6553701-dc86-4d45-a82f-f3f0ae40303e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:39.286326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 726/1430] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:39.437118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c3a35-bb69-481e-b12f-d23658117c52 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:39.438558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d39c3a35-bb69-481e-b12f-d23658117c52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:39.487328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:39.584530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39c3a35-bb69-481e-b12f-d23658117c52 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:39.697627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9ce2f9f-d9ba-4ebb-8835-ed83921948c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:39.698285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ce2f9f-d9ba-4ebb-8835-ed83921948c2 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:39.698575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ce2f9f-d9ba-4ebb-8835-ed83921948c2 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:39.698893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9ce2f9f-d9ba-4ebb-8835-ed83921948c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:39.700593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d9ce2f9f-d9ba-4ebb-8835-ed83921948c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:39.708041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:39.791110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ce2f9f-d9ba-4ebb-8835-ed83921948c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:39.792785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9ce2f9f-d9ba-4ebb-8835-ed83921948c2 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:39.811081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 705/1431] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:40.701299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a4cf9-e040-448d-a287-45fb77d71be8 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:40.701921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a4cf9-e040-448d-a287-45fb77d71be8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:40.701921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5a4cf9-e040-448d-a287-45fb77d71be8 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:40.703434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 727/1432] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:40.708292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec5868d-7d02-4ed6-8f9c-779cc85593fa None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:40.708706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec5868d-7d02-4ed6-8f9c-779cc85593fa None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:40.708859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec5868d-7d02-4ed6-8f9c-779cc85593fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:40.784452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-0ec5868d-7d02-4ed6-8f9c-779cc85593fa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:40.785642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0ec5868d-7d02-4ed6-8f9c-779cc85593fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:40.851748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:40.880099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 706/1433] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:41.031789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5a4cf9-e040-448d-a287-45fb77d71be8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:41.033478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7c5a4cf9-e040-448d-a287-45fb77d71be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:41.051391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:41.178869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5a4cf9-e040-448d-a287-45fb77d71be8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:41.265382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:41.290480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee40b0e1-efaf-49bb-9d39-0cd8adda4c44 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:41.291091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40b0e1-efaf-49bb-9d39-0cd8adda4c44 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:41.291394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40b0e1-efaf-49bb-9d39-0cd8adda4c44 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:41.291651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee40b0e1-efaf-49bb-9d39-0cd8adda4c44 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:41.293297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ee40b0e1-efaf-49bb-9d39-0cd8adda4c44 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:41.383442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee40b0e1-efaf-49bb-9d39-0cd8adda4c44 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:41.385160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee40b0e1-efaf-49bb-9d39-0cd8adda4c44 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:41.403498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 728/1434] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:42.262359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c94a6-09c6-4f3e-9edd-752f84cdc731 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:42.262967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c94a6-09c6-4f3e-9edd-752f84cdc731 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:42.262967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3c94a6-09c6-4f3e-9edd-752f84cdc731 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:42.264337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 707/1435] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:42.269371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4adffb-8d28-4243-9196-6997a0e726f5 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:42.269796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4adffb-8d28-4243-9196-6997a0e726f5 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:42.269919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4adffb-8d28-4243-9196-6997a0e726f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:42.343313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-dc4adffb-8d28-4243-9196-6997a0e726f5 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:42.344502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-dc4adffb-8d28-4243-9196-6997a0e726f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:42.386995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:42.441970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 729/1436] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:42.591219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3c94a6-09c6-4f3e-9edd-752f84cdc731 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:42.592787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5d3c94a6-09c6-4f3e-9edd-752f84cdc731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:42.741525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3c94a6-09c6-4f3e-9edd-752f84cdc731 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:42.795263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:42.854246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5541928f-10f1-41ff-979d-574544fb85c6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:42.854851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5541928f-10f1-41ff-979d-574544fb85c6 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:42.855137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5541928f-10f1-41ff-979d-574544fb85c6 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:42.855414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5541928f-10f1-41ff-979d-574544fb85c6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:42.857062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5541928f-10f1-41ff-979d-574544fb85c6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:42.944886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5541928f-10f1-41ff-979d-574544fb85c6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:42.946504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5541928f-10f1-41ff-979d-574544fb85c6 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:42.964185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 708/1437] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:43.829437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0259caf3-8bc0-4ccc-b519-6d28931abfe0 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:43.829993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0259caf3-8bc0-4ccc-b519-6d28931abfe0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:43.829993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0259caf3-8bc0-4ccc-b519-6d28931abfe0 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:43.831505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 730/1438] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:11:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:43.836310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2367f315-f908-4ac0-9097-a7ee686c1bc6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:43.836773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2367f315-f908-4ac0-9097-a7ee686c1bc6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:43.836888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2367f315-f908-4ac0-9097-a7ee686c1bc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:43.912416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2367f315-f908-4ac0-9097-a7ee686c1bc6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:43.913613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2367f315-f908-4ac0-9097-a7ee686c1bc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:43.915678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:44.011894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 709/1439] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:11:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:44.158684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0259caf3-8bc0-4ccc-b519-6d28931abfe0 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:44.160251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0259caf3-8bc0-4ccc-b519-6d28931abfe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:44.302240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0259caf3-8bc0-4ccc-b519-6d28931abfe0 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:44.328609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:44.413725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847abfa1-1c6d-49fd-92e5-d6072c0ada1f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:44.414380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847abfa1-1c6d-49fd-92e5-d6072c0ada1f service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:44.414662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847abfa1-1c6d-49fd-92e5-d6072c0ada1f service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:44.414963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847abfa1-1c6d-49fd-92e5-d6072c0ada1f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:44.416603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-847abfa1-1c6d-49fd-92e5-d6072c0ada1f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:44.505014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847abfa1-1c6d-49fd-92e5-d6072c0ada1f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:44.506616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847abfa1-1c6d-49fd-92e5-d6072c0ada1f service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:44.524332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 731/1440] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:44 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:46.452295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece988f-393c-49f3-acc1-20c1c12f5855 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:46.452836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece988f-393c-49f3-acc1-20c1c12f5855 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:46.452958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ece988f-393c-49f3-acc1-20c1c12f5855 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:46.454411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 710/1441] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:11:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:46.459645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902d857-9327-4f0a-8932-c4dbe337063d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:46.460084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902d857-9327-4f0a-8932-c4dbe337063d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:46.460206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9902d857-9327-4f0a-8932-c4dbe337063d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:46.500082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:46.525041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9902d857-9327-4f0a-8932-c4dbe337063d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:46.526307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9902d857-9327-4f0a-8932-c4dbe337063d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:46.627191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 732/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:11:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:46.664936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece988f-393c-49f3-acc1-20c1c12f5855 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:46.666476 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9ece988f-393c-49f3-acc1-20c1c12f5855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:46.769365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ece988f-393c-49f3-acc1-20c1c12f5855 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:46.801795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:46.841645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efac855-21b7-4954-9f46-38c0ce373085 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:46.842826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efac855-21b7-4954-9f46-38c0ce373085 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:46.843114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efac855-21b7-4954-9f46-38c0ce373085 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:46.843391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efac855-21b7-4954-9f46-38c0ce373085 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:46.845043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1efac855-21b7-4954-9f46-38c0ce373085 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:46.956662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efac855-21b7-4954-9f46-38c0ce373085 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:46.958104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1efac855-21b7-4954-9f46-38c0ce373085 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:46.976388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 711/1443] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:49.170920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf02e72-d87a-46e4-9fa7-23db221ef49e None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:49.171614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf02e72-d87a-46e4-9fa7-23db221ef49e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:49.171614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf02e72-d87a-46e4-9fa7-23db221ef49e None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:49.172894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 733/1444] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:11:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:49.178473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503ac9a-7481-4e7f-8f50-d11db0bdf1dc None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:49.178919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503ac9a-7481-4e7f-8f50-d11db0bdf1dc None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:49.179082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0503ac9a-7481-4e7f-8f50-d11db0bdf1dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:49.243199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-0503ac9a-7481-4e7f-8f50-d11db0bdf1dc None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:49.244411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0503ac9a-7481-4e7f-8f50-d11db0bdf1dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:49.257280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:49.343254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 712/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:11:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:49.380957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf02e72-d87a-46e4-9fa7-23db221ef49e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:49.382501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9cf02e72-d87a-46e4-9fa7-23db221ef49e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:49.484014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf02e72-d87a-46e4-9fa7-23db221ef49e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:49.496082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:49.557119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d26de6-219e-4180-bc1a-34caf83baad5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:49.557767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d26de6-219e-4180-bc1a-34caf83baad5 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:49.558047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d26de6-219e-4180-bc1a-34caf83baad5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:49.558322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d26de6-219e-4180-bc1a-34caf83baad5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:49.559977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e2d26de6-219e-4180-bc1a-34caf83baad5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:49.666045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d26de6-219e-4180-bc1a-34caf83baad5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:49.667610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d26de6-219e-4180-bc1a-34caf83baad5 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:49.685646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 734/1446] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:52.185346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5236878c-df12-43ca-9e02-174f5f4cf29d None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:52.186456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5236878c-df12-43ca-9e02-174f5f4cf29d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:52.186456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5236878c-df12-43ca-9e02-174f5f4cf29d None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:52.187855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 713/1447] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:11:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:52.193140 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377209d0-6c84-42c0-9f7b-53a5262f68aa None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:52.193600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377209d0-6c84-42c0-9f7b-53a5262f68aa None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:52.193729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377209d0-6c84-42c0-9f7b-53a5262f68aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:52.240038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:52.257511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-377209d0-6c84-42c0-9f7b-53a5262f68aa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:52.263744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-377209d0-6c84-42c0-9f7b-53a5262f68aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:52.352182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 735/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:11:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:52.391895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5236878c-df12-43ca-9e02-174f5f4cf29d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:52.393431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5236878c-df12-43ca-9e02-174f5f4cf29d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:52.497170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5236878c-df12-43ca-9e02-174f5f4cf29d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:52.537226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:52.571778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c77064-cb3d-40f2-a15a-5e6c36744c0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:52.572410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c77064-cb3d-40f2-a15a-5e6c36744c0c service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:52.572695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c77064-cb3d-40f2-a15a-5e6c36744c0c service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:52.572972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c77064-cb3d-40f2-a15a-5e6c36744c0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:52.574621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a1c77064-cb3d-40f2-a15a-5e6c36744c0c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:52.681170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c77064-cb3d-40f2-a15a-5e6c36744c0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:52.682609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c77064-cb3d-40f2-a15a-5e6c36744c0c service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:52.700952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 714/1449] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:54.844936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b29e8-f5b3-450b-9251-4fb0055aecff None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:54.845490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b29e8-f5b3-450b-9251-4fb0055aecff None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:54.845604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5b29e8-f5b3-450b-9251-4fb0055aecff None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:54.846988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 736/1450] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:11:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:54.852121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b366f7-91fb-4e2b-94b6-dc1175f0cb2f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:54.852578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b366f7-91fb-4e2b-94b6-dc1175f0cb2f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:54.852692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31b366f7-91fb-4e2b-94b6-dc1175f0cb2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:54.917441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-31b366f7-91fb-4e2b-94b6-dc1175f0cb2f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:54.918651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-31b366f7-91fb-4e2b-94b6-dc1175f0cb2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:54.939112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:55.015604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 715/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:11:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:55.053535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5b29e8-f5b3-450b-9251-4fb0055aecff None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:55.055214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4f5b29e8-f5b3-450b-9251-4fb0055aecff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:55.158650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5b29e8-f5b3-450b-9251-4fb0055aecff None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:55.178755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:55.233036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ea27de1-621b-4231-ac3b-70afb16cc80d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:55.233685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea27de1-621b-4231-ac3b-70afb16cc80d service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:55.233989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea27de1-621b-4231-ac3b-70afb16cc80d service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:55.234250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea27de1-621b-4231-ac3b-70afb16cc80d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:55.235931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5ea27de1-621b-4231-ac3b-70afb16cc80d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:55.343256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea27de1-621b-4231-ac3b-70afb16cc80d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:55.344621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ea27de1-621b-4231-ac3b-70afb16cc80d service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:55.362631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 737/1452] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:11:57.422196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a4862b-e814-4972-901e-05c59e3913c4 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:57.422706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a4862b-e814-4972-901e-05c59e3913c4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:57.422829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a4862b-e814-4972-901e-05c59e3913c4 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:57.424149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 716/1453] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:11:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:11:57.429261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c7a35-3d1a-4d14-8f01-f3050b2d1413 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:57.429701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c7a35-3d1a-4d14-8f01-f3050b2d1413 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:57.429807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f2c7a35-3d1a-4d14-8f01-f3050b2d1413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:57.492718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:57.495337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9f2c7a35-3d1a-4d14-8f01-f3050b2d1413 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:11:57.496558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9f2c7a35-3d1a-4d14-8f01-f3050b2d1413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:57.593703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 738/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:11:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:11:57.630712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a4862b-e814-4972-901e-05c59e3913c4 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:11:57.632322 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-85a4862b-e814-4972-901e-05c59e3913c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:57.739108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85a4862b-e814-4972-901e-05c59e3913c4 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:11:57.788816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:11:57.813693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d1f47-0934-4154-ac1a-7994136b0ab8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:11:57.814394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d1f47-0934-4154-ac1a-7994136b0ab8 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:11:57.814673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d1f47-0934-4154-ac1a-7994136b0ab8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:11:57.814994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d1f47-0934-4154-ac1a-7994136b0ab8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:11:57.816698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ab2d1f47-0934-4154-ac1a-7994136b0ab8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:11:57.922520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d1f47-0934-4154-ac1a-7994136b0ab8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:11:57.923979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d1f47-0934-4154-ac1a-7994136b0ab8 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:11:57.942434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 717/1455] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:11:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:00.039674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6835607f-398e-40a2-8fd8-4ad2633dc751 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:00.040214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6835607f-398e-40a2-8fd8-4ad2633dc751 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:00.040214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6835607f-398e-40a2-8fd8-4ad2633dc751 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:00.041618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 739/1456] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:12:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:00.047631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db698a-6499-4c06-a4b4-34727c9fb964 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:00.048068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db698a-6499-4c06-a4b4-34727c9fb964 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:00.048177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0db698a-6499-4c06-a4b4-34727c9fb964 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:00.114341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d0db698a-6499-4c06-a4b4-34727c9fb964 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:00.115586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d0db698a-6499-4c06-a4b4-34727c9fb964 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:00.144645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:00.211745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 718/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:12:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:00.251209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6835607f-398e-40a2-8fd8-4ad2633dc751 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:00.252808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6835607f-398e-40a2-8fd8-4ad2633dc751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:00.353842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6835607f-398e-40a2-8fd8-4ad2633dc751 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:00.382586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:00.430849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ec25a0e-0296-4929-96c8-6fd855419c84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:00.431494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec25a0e-0296-4929-96c8-6fd855419c84 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:00.431772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec25a0e-0296-4929-96c8-6fd855419c84 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:00.432045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ec25a0e-0296-4929-96c8-6fd855419c84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:00.433668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5ec25a0e-0296-4929-96c8-6fd855419c84 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:00.540573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec25a0e-0296-4929-96c8-6fd855419c84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:00.542043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ec25a0e-0296-4929-96c8-6fd855419c84 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:00.560470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 740/1458] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:02.640910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac04a390-68d1-41ea-9ac8-827a42dd42c8 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:02.641486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac04a390-68d1-41ea-9ac8-827a42dd42c8 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:02.641486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac04a390-68d1-41ea-9ac8-827a42dd42c8 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:02.642924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 719/1459] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:12:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:02.648073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03164cf7-dd10-43e9-b4b3-5c0757363064 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:02.648524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03164cf7-dd10-43e9-b4b3-5c0757363064 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:02.648636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03164cf7-dd10-43e9-b4b3-5c0757363064 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:02.713903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-03164cf7-dd10-43e9-b4b3-5c0757363064 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:02.715155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-03164cf7-dd10-43e9-b4b3-5c0757363064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:02.724431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:02.810784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 741/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:12:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:02.848602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac04a390-68d1-41ea-9ac8-827a42dd42c8 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:02.850164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ac04a390-68d1-41ea-9ac8-827a42dd42c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:03.072708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac04a390-68d1-41ea-9ac8-827a42dd42c8 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:03.128868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:03.147483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc292e70-2952-4a2a-bf70-ca05bb5bb888 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:03.148244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc292e70-2952-4a2a-bf70-ca05bb5bb888 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:03.148626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc292e70-2952-4a2a-bf70-ca05bb5bb888 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:03.149003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc292e70-2952-4a2a-bf70-ca05bb5bb888 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:03.150746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dc292e70-2952-4a2a-bf70-ca05bb5bb888 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:03.258594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc292e70-2952-4a2a-bf70-ca05bb5bb888 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:03.260083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc292e70-2952-4a2a-bf70-ca05bb5bb888 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:03.278444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 720/1461] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 430 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:05.381121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cc6631-e231-4112-886e-384d3a486a76 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:05.381693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cc6631-e231-4112-886e-384d3a486a76 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:05.381693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1cc6631-e231-4112-886e-384d3a486a76 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:05.383218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 742/1462] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:12:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:12:05.388785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f098147b-a709-4e1d-960f-9fc0eb36885a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:05.389301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f098147b-a709-4e1d-960f-9fc0eb36885a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:05.389457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f098147b-a709-4e1d-960f-9fc0eb36885a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:05.453654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f098147b-a709-4e1d-960f-9fc0eb36885a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:05.454913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f098147b-a709-4e1d-960f-9fc0eb36885a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:05.492909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:05.552256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 721/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:12:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:05.589444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cc6631-e231-4112-886e-384d3a486a76 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:05.591134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a1cc6631-e231-4112-886e-384d3a486a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:05.694002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1cc6631-e231-4112-886e-384d3a486a76 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:05.732378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:05.769306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6169ad9b-0a11-4d9d-a691-edaba0286bf2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:05.769956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169ad9b-0a11-4d9d-a691-edaba0286bf2 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:05.770244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169ad9b-0a11-4d9d-a691-edaba0286bf2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:05.770524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6169ad9b-0a11-4d9d-a691-edaba0286bf2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:05.772220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6169ad9b-0a11-4d9d-a691-edaba0286bf2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:05.881584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6169ad9b-0a11-4d9d-a691-edaba0286bf2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:05.883072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6169ad9b-0a11-4d9d-a691-edaba0286bf2 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:05.902278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 743/1464] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:07.921079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4667261f-29d4-4db4-b8de-a1e1816f9554 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:07.921656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4667261f-29d4-4db4-b8de-a1e1816f9554 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:07.921757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4667261f-29d4-4db4-b8de-a1e1816f9554 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:07.923110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 722/1465] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:12:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:07.928694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9ff791-d654-4246-adaa-72e13f740d2e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:07.929121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9ff791-d654-4246-adaa-72e13f740d2e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:07.929293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9ff791-d654-4246-adaa-72e13f740d2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:07.995964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-da9ff791-d654-4246-adaa-72e13f740d2e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:07.997188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-da9ff791-d654-4246-adaa-72e13f740d2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:08.015297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:08.094105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 744/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:12:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:08.131506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4667261f-29d4-4db4-b8de-a1e1816f9554 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:08.133021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4667261f-29d4-4db4-b8de-a1e1816f9554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:08.242344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4667261f-29d4-4db4-b8de-a1e1816f9554 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:08.308594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:08.315734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25f5cdc-332f-48ea-8315-549eef638ad4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:08.316373 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25f5cdc-332f-48ea-8315-549eef638ad4 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:08.316652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25f5cdc-332f-48ea-8315-549eef638ad4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:08.316920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25f5cdc-332f-48ea-8315-549eef638ad4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:08.318540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a25f5cdc-332f-48ea-8315-549eef638ad4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:08.431940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25f5cdc-332f-48ea-8315-549eef638ad4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:08.433417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25f5cdc-332f-48ea-8315-549eef638ad4 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:08.452218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 723/1467] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:10.504252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77c0932-a959-4693-819f-3df9ce306e32 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:10.504786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77c0932-a959-4693-819f-3df9ce306e32 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:10.504786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f77c0932-a959-4693-819f-3df9ce306e32 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:10.506118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 745/1468] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:12:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:10.511648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca6e1ce-09d9-4364-8fc6-dd2a148b8369 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:10.512125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca6e1ce-09d9-4364-8fc6-dd2a148b8369 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:10.512229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca6e1ce-09d9-4364-8fc6-dd2a148b8369 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:10.577698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-aca6e1ce-09d9-4364-8fc6-dd2a148b8369 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:10.578943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-aca6e1ce-09d9-4364-8fc6-dd2a148b8369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:10.627280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:10.675525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 724/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:12:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:10.711665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77c0932-a959-4693-819f-3df9ce306e32 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:10.713275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f77c0932-a959-4693-819f-3df9ce306e32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:10.818082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f77c0932-a959-4693-819f-3df9ce306e32 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:10.865296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:10.891052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f9caaec-8628-4aae-ac55-121de0828c49 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:10.891707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9caaec-8628-4aae-ac55-121de0828c49 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:10.891994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9caaec-8628-4aae-ac55-121de0828c49 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:10.892277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f9caaec-8628-4aae-ac55-121de0828c49 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:10.893937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1f9caaec-8628-4aae-ac55-121de0828c49 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:11.003490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9caaec-8628-4aae-ac55-121de0828c49 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:11.004918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f9caaec-8628-4aae-ac55-121de0828c49 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:11.023639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 746/1470] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:20.673687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=22 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:21.021266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=66 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:31.501471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788610ec-025e-4da7-a2b1-906b28eb37ea None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:31.502046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788610ec-025e-4da7-a2b1-906b28eb37ea None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:31.502046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788610ec-025e-4da7-a2b1-906b28eb37ea None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:31.503421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 725/1471] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:12:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:31.508523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2e865-6b0a-471e-be8d-8e6e31a83188 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:31.508960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2e865-6b0a-471e-be8d-8e6e31a83188 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:31.509080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28b2e865-6b0a-471e-be8d-8e6e31a83188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:31.573278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-28b2e865-6b0a-471e-be8d-8e6e31a83188 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:31.574498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-28b2e865-6b0a-471e-be8d-8e6e31a83188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:31.670156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 747/1472] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:12:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:31.754927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92409ba0-eb17-45df-a28b-a35675227f2b None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:31.755155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92409ba0-eb17-45df-a28b-a35675227f2b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:31.755431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92409ba0-eb17-45df-a28b-a35675227f2b None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:31.756787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 726/1473] 10.222.0.27 () {54 vars in 962 bytes} [Thu Jul 14 12:12:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:31.761498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d35cb4-27a0-4c2c-b52b-65a800bf5fd0 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:31.761791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d35cb4-27a0-4c2c-b52b-65a800bf5fd0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:31.762072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d35cb4-27a0-4c2c-b52b-65a800bf5fd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:31.836345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a3d35cb4-27a0-4c2c-b52b-65a800bf5fd0 None None] MFA Rules not processed for user `86ace7e877e3484b8853536b77c76885`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:31.837553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a3d35cb4-27a0-4c2c-b52b-65a800bf5fd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:31.857978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:31.913069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 748/1474] 10.222.0.27 () {58 vars in 1059 bytes} [Thu Jul 14 12:12:31 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:31.921008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92409ba0-eb17-45df-a28b-a35675227f2b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:31.922632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-92409ba0-eb17-45df-a28b-a35675227f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:32.029491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92409ba0-eb17-45df-a28b-a35675227f2b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:32.101813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feae018b-91bf-49c1-8554-3872f92385ee service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:32.102462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feae018b-91bf-49c1-8554-3872f92385ee service glance] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:32.102745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feae018b-91bf-49c1-8554-3872f92385ee service glance] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:32.103060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feae018b-91bf-49c1-8554-3872f92385ee service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:32.104715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-feae018b-91bf-49c1-8554-3872f92385ee service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:32.161469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:32.216410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feae018b-91bf-49c1-8554-3872f92385ee service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:32.217889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feae018b-91bf-49c1-8554-3872f92385ee service glance] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:32.236341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 727/1475] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:12:34.363596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff72f48-0615-4c0f-9e64-62507c6043ed None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:34.364167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff72f48-0615-4c0f-9e64-62507c6043ed None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:34.364167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff72f48-0615-4c0f-9e64-62507c6043ed None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:34.365548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 749/1476] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:12:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:34.370653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74826d54-2c32-4279-a1da-592dc3133d54 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:34.371141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74826d54-2c32-4279-a1da-592dc3133d54 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:34.371282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74826d54-2c32-4279-a1da-592dc3133d54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:34.436094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-74826d54-2c32-4279-a1da-592dc3133d54 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:34.437313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-74826d54-2c32-4279-a1da-592dc3133d54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:34.533560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 728/1477] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:12:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:34.557017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff72f48-0615-4c0f-9e64-62507c6043ed None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:34.558581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5ff72f48-0615-4c0f-9e64-62507c6043ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:34.683536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff72f48-0615-4c0f-9e64-62507c6043ed None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:34.732458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:34.777231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00dddaf9-b808-4704-99fd-831a7b35731d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:34.777895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dddaf9-b808-4704-99fd-831a7b35731d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:34.778180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dddaf9-b808-4704-99fd-831a7b35731d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:34.778491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00dddaf9-b808-4704-99fd-831a7b35731d admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:34.779282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dddaf9-b808-4704-99fd-831a7b35731d admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:34.781044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-00dddaf9-b808-4704-99fd-831a7b35731d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:12:34.781044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:12:34.781044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:12:34.782200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00dddaf9-b808-4704-99fd-831a7b35731d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:34.801174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 750/1478] 10.222.0.27 () {56 vars in 1108 bytes} [Thu Jul 14 12:12:34 2022] GET /identity/v3/projects => generated 2553 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:12:36.837109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbb5d95-e36f-466e-aad6-f698d82d2d39 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:36.837643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbb5d95-e36f-466e-aad6-f698d82d2d39 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:36.837643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbb5d95-e36f-466e-aad6-f698d82d2d39 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:36.839097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 729/1479] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:12:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:36.844608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef66d8d-e29d-44c5-9d61-f34b531449f5 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:36.845049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef66d8d-e29d-44c5-9d61-f34b531449f5 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:36.845154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef66d8d-e29d-44c5-9d61-f34b531449f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:36.909898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1ef66d8d-e29d-44c5-9d61-f34b531449f5 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:36.911145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1ef66d8d-e29d-44c5-9d61-f34b531449f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:37.005646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 751/1480] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:12:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:37.048592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbb5d95-e36f-466e-aad6-f698d82d2d39 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:37.050089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9bbb5d95-e36f-466e-aad6-f698d82d2d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:37.060787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:37.196606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbb5d95-e36f-466e-aad6-f698d82d2d39 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:37.275609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:37.310366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd69895-0092-4913-b053-86df00621134 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:37.311047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd69895-0092-4913-b053-86df00621134 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:37.311334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd69895-0092-4913-b053-86df00621134 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:37.311621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd69895-0092-4913-b053-86df00621134 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:37.313312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2dd69895-0092-4913-b053-86df00621134 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:37.400907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd69895-0092-4913-b053-86df00621134 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:37.402508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd69895-0092-4913-b053-86df00621134 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:37.420993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 730/1481] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:39.632954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbaaa85-7814-4d8f-b19d-3edae7ad1a30 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:39.633441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbaaa85-7814-4d8f-b19d-3edae7ad1a30 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:39.633537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbaaa85-7814-4d8f-b19d-3edae7ad1a30 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:39.635145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 752/1482] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:12:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:39.640970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2c4fe6-9b00-4c53-adb9-754bd5afe41f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:39.641406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2c4fe6-9b00-4c53-adb9-754bd5afe41f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:39.641533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2c4fe6-9b00-4c53-adb9-754bd5afe41f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:39.705652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-be2c4fe6-9b00-4c53-adb9-754bd5afe41f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:39.706921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-be2c4fe6-9b00-4c53-adb9-754bd5afe41f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:39.760829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:39.813111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 731/1483] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:12:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:39.848814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbaaa85-7814-4d8f-b19d-3edae7ad1a30 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:39.850372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4dbaaa85-7814-4d8f-b19d-3edae7ad1a30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:39.874162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:39.952750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbaaa85-7814-4d8f-b19d-3edae7ad1a30 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:40.026817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a438ec5-1cba-4714-bf49-5e830b739d6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:40.027486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a438ec5-1cba-4714-bf49-5e830b739d6b service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:40.027769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a438ec5-1cba-4714-bf49-5e830b739d6b service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:40.028047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a438ec5-1cba-4714-bf49-5e830b739d6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:40.029755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2a438ec5-1cba-4714-bf49-5e830b739d6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:40.223982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:40.260540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a438ec5-1cba-4714-bf49-5e830b739d6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:40.262020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a438ec5-1cba-4714-bf49-5e830b739d6b service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:40.282195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 753/1484] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 434 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:42.384405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3b4a07-72ba-4a05-8727-ff26e94589ea None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:42.384942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3b4a07-72ba-4a05-8727-ff26e94589ea None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:42.384942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a3b4a07-72ba-4a05-8727-ff26e94589ea None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:42.386376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 732/1485] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:12:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:42.391744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e3f6d3-dc62-4506-a600-4c81b555d427 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:42.392201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e3f6d3-dc62-4506-a600-4c81b555d427 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:42.392331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e3f6d3-dc62-4506-a600-4c81b555d427 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:42.456674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f4e3f6d3-dc62-4506-a600-4c81b555d427 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:42.457900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f4e3f6d3-dc62-4506-a600-4c81b555d427 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:42.552663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 754/1486] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:12:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:42.589881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3b4a07-72ba-4a05-8727-ff26e94589ea None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:42.592333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8a3b4a07-72ba-4a05-8727-ff26e94589ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:42.697487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a3b4a07-72ba-4a05-8727-ff26e94589ea None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:42.770292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:42.772114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb1197c8-4075-4dc3-b01a-3056587f82a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:42.772760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1197c8-4075-4dc3-b01a-3056587f82a1 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:42.773448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1197c8-4075-4dc3-b01a-3056587f82a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:42.773448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb1197c8-4075-4dc3-b01a-3056587f82a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:42.774954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fb1197c8-4075-4dc3-b01a-3056587f82a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:42.889246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1197c8-4075-4dc3-b01a-3056587f82a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:42.891207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb1197c8-4075-4dc3-b01a-3056587f82a1 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:42.910387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 733/1487] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:45.043559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac246d43-1bdb-431b-888a-4907192097e1 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:45.044620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac246d43-1bdb-431b-888a-4907192097e1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:45.044620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac246d43-1bdb-431b-888a-4907192097e1 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:45.045964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 755/1488] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:12:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:45.051167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688ae602-33ae-42f4-bd98-4164d1afb16e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:45.051578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688ae602-33ae-42f4-bd98-4164d1afb16e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:45.051728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688ae602-33ae-42f4-bd98-4164d1afb16e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:45.115745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-688ae602-33ae-42f4-bd98-4164d1afb16e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:45.116961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-688ae602-33ae-42f4-bd98-4164d1afb16e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:45.167968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:45.211559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 734/1489] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:12:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:45.253309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac246d43-1bdb-431b-888a-4907192097e1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:45.254837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ac246d43-1bdb-431b-888a-4907192097e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:45.290123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:45.356945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac246d43-1bdb-431b-888a-4907192097e1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:45.439134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136c8362-8945-4a46-9dd6-fb69d786652b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:45.439780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c8362-8945-4a46-9dd6-fb69d786652b service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:45.440055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c8362-8945-4a46-9dd6-fb69d786652b service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:45.440322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136c8362-8945-4a46-9dd6-fb69d786652b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:45.441960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-136c8362-8945-4a46-9dd6-fb69d786652b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:45.519966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:45.550830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136c8362-8945-4a46-9dd6-fb69d786652b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:45.552298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136c8362-8945-4a46-9dd6-fb69d786652b service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:45.570695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 756/1490] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:47.893313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04348b99-3bef-4d14-99a0-b5b9cab4090f None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:47.893851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04348b99-3bef-4d14-99a0-b5b9cab4090f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:47.893851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04348b99-3bef-4d14-99a0-b5b9cab4090f None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:47.895323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 735/1491] 10.222.0.27 () {54 vars in 926 bytes} [Thu Jul 14 12:12:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:12:47.900550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5acb3-a846-41e7-83ac-b40661975638 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:47.900996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5acb3-a846-41e7-83ac-b40661975638 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:47.901111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5acb3-a846-41e7-83ac-b40661975638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:47.963973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e3c5acb3-a846-41e7-83ac-b40661975638 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:12:47.965159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e3c5acb3-a846-41e7-83ac-b40661975638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:48.060602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 757/1492] 10.222.0.27 () {58 vars in 1023 bytes} [Thu Jul 14 12:12:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:12:48.099759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04348b99-3bef-4d14-99a0-b5b9cab4090f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:12:48.101308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-04348b99-3bef-4d14-99a0-b5b9cab4090f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:48.204532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04348b99-3bef-4d14-99a0-b5b9cab4090f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:12:48.278034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81ebeab6-02e8-4a05-a000-32d992ba2e17 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:12:48.279014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ebeab6-02e8-4a05-a000-32d992ba2e17 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:12:48.279296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ebeab6-02e8-4a05-a000-32d992ba2e17 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:12:48.279572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81ebeab6-02e8-4a05-a000-32d992ba2e17 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:12:48.281209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-81ebeab6-02e8-4a05-a000-32d992ba2e17 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:12:48.290379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:48.391926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ebeab6-02e8-4a05-a000-32d992ba2e17 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:12:48.393357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81ebeab6-02e8-4a05-a000-32d992ba2e17 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:12:48.411720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 736/1493] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:12:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:12:58.058195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=82 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:12:58.409110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=50 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:24.968080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:24.968680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:24.968680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:25.041483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:14:25.042716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:25.135524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 758/1494] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:14:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:14:25.143688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7324196b-7fb8-487d-8089-4d8e990356c6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:25.144188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7324196b-7fb8-487d-8089-4d8e990356c6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:25.144298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7324196b-7fb8-487d-8089-4d8e990356c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:25.219087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7324196b-7fb8-487d-8089-4d8e990356c6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:14:25.220299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7324196b-7fb8-487d-8089-4d8e990356c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:25.315388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 737/1495] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:14:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:14:25.320068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:25.321304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:25.333482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0844acc9-4719-4c52-8c61-97e67f050677 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:25.394182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:25.427351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:25.428009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:25.428290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:25.428576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:25.429272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:14:25.429575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:25.430592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:25.430592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:14:25.430592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:14:25.431525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:25.443059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 759/1496] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:14:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:14:25.452026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7324196b-7fb8-487d-8089-4d8e990356c6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:25.461170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7324196b-7fb8-487d-8089-4d8e990356c6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:25.521430 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:25.556916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:25.557588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:25.558485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:25.558795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:25.559490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:25.561256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING py.warnings [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:25.561256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: warnings.warn(msg) Jul 14 12:14:25.561256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: Jul 14 12:14:25.562450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:25.692588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 738/1497] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:14:25 2022] POST /identity/v3/projects => generated 377 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:14:25.696747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:25.698280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:25.734230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9679f893-9e97-4687-9602-1c97f22ced55 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:25.825250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:25.825914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:25.826197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:25.826471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:25.827179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:25.828194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:25.828194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:14:25.828194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:14:25.829385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:25.932226 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 760/1498] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:14:25 2022] POST /identity/v3/users => generated 289 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:14:25.936537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:25.938052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:25.999724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef369a88-c197-4d84-9ab8-0a17850ac01f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:26.083144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:26.093141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:26.095388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:26.095388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:26.095388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:26.095388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:26.096130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING py.warnings [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:26.096130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: warnings.warn(msg) Jul 14 12:14:26.096130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: Jul 14 12:14:26.097017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:26.109650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 739/1499] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:14:26.113929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:26.115467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:26.127234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cd1c935-adc8-49ab-967f-6b4a9c36a6b3 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:26.151777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:26.220739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0f9dc42-b91e-4ca7-bc6c-1fd09a23c8d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:26.221395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9dc42-b91e-4ca7-bc6c-1fd09a23c8d6 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:26.221671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9dc42-b91e-4ca7-bc6c-1fd09a23c8d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:26.221944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0f9dc42-b91e-4ca7-bc6c-1fd09a23c8d6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:26.222660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f9dc42-b91e-4ca7-bc6c-1fd09a23c8d6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:26.223691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-f0f9dc42-b91e-4ca7-bc6c-1fd09a23c8d6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:26.223691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:14:26.223691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:14:26.224556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0f9dc42-b91e-4ca7-bc6c-1fd09a23c8d6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:26.237165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 761/1500] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:14:26 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:14:26.241297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:26.242884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:26.276106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffde1e1-c2eb-46f0-96d3-5c4de48333b3 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:26.369878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f40bfff6-4731-48af-b7af-b0716e3169ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:26.371327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40bfff6-4731-48af-b7af-b0716e3169ad admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:26.371612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40bfff6-4731-48af-b7af-b0716e3169ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:26.371888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f40bfff6-4731-48af-b7af-b0716e3169ad admin admin] PATH_INFO: `/v3/projects/13a92d869e4b42359851e8741827b8d0/users/7b22539ea4d54a59a40a1e382684136c/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:26.394141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40bfff6-4731-48af-b7af-b0716e3169ad admin admin] RBAC: Authorizing `identity:create_grant(project_id=13a92d869e4b42359851e8741827b8d0, user_id=7b22539ea4d54a59a40a1e382684136c, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:26.395199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING py.warnings [None req-f40bfff6-4731-48af-b7af-b0716e3169ad admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:26.395199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: warnings.warn(msg) Jul 14 12:14:26.395199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: Jul 14 12:14:26.397119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f40bfff6-4731-48af-b7af-b0716e3169ad admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:26.483716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 740/1501] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:14:26 2022] PUT /identity/v3/projects/13a92d869e4b42359851e8741827b8d0/users/7b22539ea4d54a59a40a1e382684136c/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:14:26.496263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c483e165-d631-400c-a828-437edda68ceb None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:26.496679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c483e165-d631-400c-a828-437edda68ceb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:26.496968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c483e165-d631-400c-a828-437edda68ceb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:26.552279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c483e165-d631-400c-a828-437edda68ceb None None] MFA Rules not processed for user `7b22539ea4d54a59a40a1e382684136c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:14:26.553428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c483e165-d631-400c-a828-437edda68ceb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:26.633641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 762/1502] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:14:26 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:14:26.647376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadede9c-80e1-4428-a3c3-66355e62f4f1 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:26.647689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadede9c-80e1-4428-a3c3-66355e62f4f1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:26.647960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cadede9c-80e1-4428-a3c3-66355e62f4f1 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:26.649319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 741/1503] 10.222.0.27 () {54 vars in 963 bytes} [Thu Jul 14 12:14:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:14:26.654269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdb9a77-df35-4943-865d-37a885bb0a27 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:26.654561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdb9a77-df35-4943-865d-37a885bb0a27 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:26.654905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdb9a77-df35-4943-865d-37a885bb0a27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:26.686479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:26.736595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9bdb9a77-df35-4943-865d-37a885bb0a27 None None] MFA Rules not processed for user `e114b292a7f544b0b24016199edcef25`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:14:26.738045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9bdb9a77-df35-4943-865d-37a885bb0a27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:26.814853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 763/1504] 10.222.0.27 () {58 vars in 1060 bytes} [Thu Jul 14 12:14:26 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:14:26.822352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadede9c-80e1-4428-a3c3-66355e62f4f1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:26.824364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cadede9c-80e1-4428-a3c3-66355e62f4f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:26.876184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:26.929890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cadede9c-80e1-4428-a3c3-66355e62f4f1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:27.002560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:27.003246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:27.003531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:27.003806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:27.005456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:27.108996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:27.110357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:27.113870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:27.128134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 742/1505] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:14:26 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:14:27.172821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da12966-c7fb-480e-94c1-bc67ad1602dc None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:27.173276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da12966-c7fb-480e-94c1-bc67ad1602dc None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:27.173382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da12966-c7fb-480e-94c1-bc67ad1602dc None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:27.174690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 764/1506] 10.222.0.27 () {52 vars in 829 bytes} [Thu Jul 14 12:14:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:14:27.178220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:27.179544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:27.276358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d72de8f-d308-4e2a-ab72-3ab13dbd3759 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:27.370579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:27.371497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:27.371783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:27.372063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] PATH_INFO: `/v3/users/7b22539ea4d54a59a40a1e382684136c` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:27.384378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b22539ea4d54a59a40a1e382684136c)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:27.385405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING py.warnings [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:27.385405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: warnings.warn(msg) Jul 14 12:14:27.385405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: Jul 14 12:14:27.387209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:27.396787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:27.600124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b22539ea4d54a59a40a1e382684136c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:27.684575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b22539ea4d54a59a40a1e382684136c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:27.684936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b22539ea4d54a59a40a1e382684136c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:27.697260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b22539ea4d54a59a40a1e382684136c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:27.697794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b22539ea4d54a59a40a1e382684136c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:27.723387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 743/1507] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:14:27 2022] DELETE /identity/v3/users/7b22539ea4d54a59a40a1e382684136c => generated 0 bytes in 546 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:14:27.727841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:27.728125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:27.728394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:27.784108 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:14:27.785306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:27.858256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:27.871035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 765/1508] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:14:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:14:27.880597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:27.881891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:27.980635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b5c5d7-8dbb-4b69-a5e2-86098b275845 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:28.033837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:28.052285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c6787e-9077-4cd1-a291-8edb1995ec2e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:28.052931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c6787e-9077-4cd1-a291-8edb1995ec2e service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:28.053224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c6787e-9077-4cd1-a291-8edb1995ec2e service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:28.053493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c6787e-9077-4cd1-a291-8edb1995ec2e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:28.055151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d1c6787e-9077-4cd1-a291-8edb1995ec2e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:28.167742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c6787e-9077-4cd1-a291-8edb1995ec2e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:28.169203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c6787e-9077-4cd1-a291-8edb1995ec2e service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:28.189292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 744/1509] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:14:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:14:28.722295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:14:28.723631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:14:28.736228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb80e887-e9d8-45aa-b55e-5ab2330a27b1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:14:28.835599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:14:28.836514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:14:28.836719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:14:28.836989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] PATH_INFO: `/v3/projects/13a92d869e4b42359851e8741827b8d0` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:14:28.847017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] RBAC: Authorizing `identity:delete_project(project_id=13a92d869e4b42359851e8741827b8d0)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:14:28.848041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:14:28.848041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:14:28.848041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:14:28.849229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:14:29.049252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '13a92d869e4b42359851e8741827b8d0', 'request_id': 'req-9c573d32-953f-4b5e-8001-fcb6a6327eaf'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:29.125332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '13a92d869e4b42359851e8741827b8d0', 'request_id': 'req-9c573d32-953f-4b5e-8001-fcb6a6327eaf'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:29.139216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '13a92d869e4b42359851e8741827b8d0', 'request_id': 'req-9c573d32-953f-4b5e-8001-fcb6a6327eaf'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:29.163262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] The token cache is being invalidate because project 13a92d869e4b42359851e8741827b8d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:14:29.163551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:29.164079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9c573d32-953f-4b5e-8001-fcb6a6327eaf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:14:29.165481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 766/1510] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:14:28 2022] DELETE /identity/v3/projects/13a92d869e4b42359851e8741827b8d0 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:14:38.187811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=62 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:14:39.163608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:16:12.157524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b55791-a6c1-42a3-856c-a6fc0814cb57 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:12.158815 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b55791-a6c1-42a3-856c-a6fc0814cb57 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:12.158815 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b55791-a6c1-42a3-856c-a6fc0814cb57 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:12.160082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 745/1511] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:16:12 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:12.165157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca874ba2-bf98-4717-8a0d-b94bfbdb68a7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:12.165715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca874ba2-bf98-4717-8a0d-b94bfbdb68a7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:12.165715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca874ba2-bf98-4717-8a0d-b94bfbdb68a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:12.243616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ca874ba2-bf98-4717-8a0d-b94bfbdb68a7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:12.244840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ca874ba2-bf98-4717-8a0d-b94bfbdb68a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:12.347996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 767/1512] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:16:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:12.498414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b55791-a6c1-42a3-856c-a6fc0814cb57 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:12.500571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-09b55791-a6c1-42a3-856c-a6fc0814cb57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:12.620357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b55791-a6c1-42a3-856c-a6fc0814cb57 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:12.625774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d31a591d-3e95-4662-b06d-5fd732b674b3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:12.626426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a591d-3e95-4662-b06d-5fd732b674b3 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:12.627346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a591d-3e95-4662-b06d-5fd732b674b3 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:12.627628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d31a591d-3e95-4662-b06d-5fd732b674b3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:12.635945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a591d-3e95-4662-b06d-5fd732b674b3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:12.637569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d31a591d-3e95-4662-b06d-5fd732b674b3 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:12.647246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 746/1513] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:13.503999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6dff9-0c33-45bd-84d4-0ce4d0928bda None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:13.504582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6dff9-0c33-45bd-84d4-0ce4d0928bda None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:13.504582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb6dff9-0c33-45bd-84d4-0ce4d0928bda None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:13.506022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 768/1514] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:16:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:13.510823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70676789-2494-48e8-8b9f-8ab60cd3fc37 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:13.511296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70676789-2494-48e8-8b9f-8ab60cd3fc37 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:13.511411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70676789-2494-48e8-8b9f-8ab60cd3fc37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:13.529453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-70676789-2494-48e8-8b9f-8ab60cd3fc37 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:13.530656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-70676789-2494-48e8-8b9f-8ab60cd3fc37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:13.539017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 747/1515] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:16:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:13.604061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fca39c-8134-4eee-bd7a-52134a41e0ec None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:13.604375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fca39c-8134-4eee-bd7a-52134a41e0ec None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:13.604662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fca39c-8134-4eee-bd7a-52134a41e0ec None None] PATH_INFO: `/v3` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:13.606021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 769/1516] 10.222.0.86 () {54 vars in 925 bytes} [Thu Jul 14 12:16:13 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:16:13.613109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad37b09-bb08-4add-b644-25c3646babff None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:13.613395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad37b09-bb08-4add-b644-25c3646babff None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:13.613671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ad37b09-bb08-4add-b644-25c3646babff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:13.671509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7ad37b09-bb08-4add-b644-25c3646babff None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:13.672722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7ad37b09-bb08-4add-b644-25c3646babff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:13.703047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fca39c-8134-4eee-bd7a-52134a41e0ec None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:13.718714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fca39c-8134-4eee-bd7a-52134a41e0ec None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:13.725371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99202da-df68-4087-b6b6-3111cc06dcd6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:13.726021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99202da-df68-4087-b6b6-3111cc06dcd6 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:13.726301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99202da-df68-4087-b6b6-3111cc06dcd6 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:13.726626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99202da-df68-4087-b6b6-3111cc06dcd6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:13.735322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99202da-df68-4087-b6b6-3111cc06dcd6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:13.735819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:16:13.737958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99202da-df68-4087-b6b6-3111cc06dcd6 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:13.747678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 770/1517] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:16:13.755609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 748/1518] 10.222.0.86 () {58 vars in 1015 bytes} [Thu Jul 14 12:16:13 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:23.393315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37917e-3c3b-4045-99a5-d9f4fcc3a8a1 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:23.393886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37917e-3c3b-4045-99a5-d9f4fcc3a8a1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:23.393886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c37917e-3c3b-4045-99a5-d9f4fcc3a8a1 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:23.395211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 771/1519] 10.222.0.27 () {54 vars in 960 bytes} [Thu Jul 14 12:16:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:23.400232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5f659-7439-4587-aa17-9b577a0c1b47 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:23.400662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5f659-7439-4587-aa17-9b577a0c1b47 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:23.400809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85a5f659-7439-4587-aa17-9b577a0c1b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:23.419644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-85a5f659-7439-4587-aa17-9b577a0c1b47 None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:23.420845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-85a5f659-7439-4587-aa17-9b577a0c1b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:23.430115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 749/1520] 10.222.0.27 () {58 vars in 1057 bytes} [Thu Jul 14 12:16:23 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:23.437670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37917e-3c3b-4045-99a5-d9f4fcc3a8a1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:23.451753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c37917e-3c3b-4045-99a5-d9f4fcc3a8a1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:23.456681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d59f4609-9902-4b52-9ddd-e30c81f04f29 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:23.457314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59f4609-9902-4b52-9ddd-e30c81f04f29 service placement] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:23.457601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59f4609-9902-4b52-9ddd-e30c81f04f29 service placement] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:23.457881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d59f4609-9902-4b52-9ddd-e30c81f04f29 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:23.466017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59f4609-9902-4b52-9ddd-e30c81f04f29 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:23.467631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d59f4609-9902-4b52-9ddd-e30c81f04f29 service placement] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:23.477628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 772/1521] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:23 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:24.984092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42272380-57b0-4c7a-910f-2cec3be89d77 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:24.984649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42272380-57b0-4c7a-910f-2cec3be89d77 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:24.984649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42272380-57b0-4c7a-910f-2cec3be89d77 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:24.986565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 750/1522] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:16:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:24.991349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69949669-d248-485a-b6ae-d78001b6e216 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:24.991793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69949669-d248-485a-b6ae-d78001b6e216 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:24.991897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69949669-d248-485a-b6ae-d78001b6e216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:25.010651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-69949669-d248-485a-b6ae-d78001b6e216 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:25.011889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-69949669-d248-485a-b6ae-d78001b6e216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:25.020287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 773/1523] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:16:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:25.175824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42272380-57b0-4c7a-910f-2cec3be89d77 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:25.190058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42272380-57b0-4c7a-910f-2cec3be89d77 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:25.195547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c125f1af-8551-4191-b8b0-236b7c9f6497 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:25.196184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c125f1af-8551-4191-b8b0-236b7c9f6497 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:25.196466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c125f1af-8551-4191-b8b0-236b7c9f6497 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:25.196739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c125f1af-8551-4191-b8b0-236b7c9f6497 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:25.205298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c125f1af-8551-4191-b8b0-236b7c9f6497 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:25.206919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c125f1af-8551-4191-b8b0-236b7c9f6497 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:25.216342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 751/1524] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:26.033800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e15cb4-dea5-45ad-b3b6-a3b46671b337 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:26.034348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e15cb4-dea5-45ad-b3b6-a3b46671b337 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:26.034348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e15cb4-dea5-45ad-b3b6-a3b46671b337 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:26.035708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 774/1525] 127.0.0.1 () {54 vars in 901 bytes} [Thu Jul 14 12:16:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:26.041273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d721192e-4055-439c-a15b-021a876b2858 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:26.041706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d721192e-4055-439c-a15b-021a876b2858 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:26.041825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d721192e-4055-439c-a15b-021a876b2858 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:26.060084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d721192e-4055-439c-a15b-021a876b2858 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:26.061275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d721192e-4055-439c-a15b-021a876b2858 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:26.069997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 752/1526] 127.0.0.1 () {58 vars in 998 bytes} [Thu Jul 14 12:16:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:26.219587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e15cb4-dea5-45ad-b3b6-a3b46671b337 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:26.233845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e15cb4-dea5-45ad-b3b6-a3b46671b337 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:26.240450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f2271f5-ee9d-4543-9e4f-b09cc06a5ea8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:26.241081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2271f5-ee9d-4543-9e4f-b09cc06a5ea8 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:26.241365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2271f5-ee9d-4543-9e4f-b09cc06a5ea8 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:26.241643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f2271f5-ee9d-4543-9e4f-b09cc06a5ea8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:26.250044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2271f5-ee9d-4543-9e4f-b09cc06a5ea8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:26.252522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f2271f5-ee9d-4543-9e4f-b09cc06a5ea8 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:26.261952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 775/1527] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:30.146367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617f689-3e1b-4fa7-911c-06e53af89058 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:30.146936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617f689-3e1b-4fa7-911c-06e53af89058 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:30.146936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3617f689-3e1b-4fa7-911c-06e53af89058 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:30.148244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 753/1528] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:16:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:30.153661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a34af-a5ad-4302-9f56-705251cc1b09 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:30.161210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a34af-a5ad-4302-9f56-705251cc1b09 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:30.161210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86a34af-a5ad-4302-9f56-705251cc1b09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:30.172516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c86a34af-a5ad-4302-9f56-705251cc1b09 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:30.174412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c86a34af-a5ad-4302-9f56-705251cc1b09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:30.182282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 776/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:16:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:30.256647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3617f689-3e1b-4fa7-911c-06e53af89058 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:30.258318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3617f689-3e1b-4fa7-911c-06e53af89058 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:30.336990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3617f689-3e1b-4fa7-911c-06e53af89058 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:30.340573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85489042-b1c6-4d65-86cd-388348f39d7e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:30.341217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85489042-b1c6-4d65-86cd-388348f39d7e service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:30.341501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85489042-b1c6-4d65-86cd-388348f39d7e service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:30.341783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85489042-b1c6-4d65-86cd-388348f39d7e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:30.351294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85489042-b1c6-4d65-86cd-388348f39d7e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:30.352725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85489042-b1c6-4d65-86cd-388348f39d7e service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:30.362782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 754/1530] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:32.375191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d20b3-d024-4417-8a09-de7681eb8476 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:32.375758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d20b3-d024-4417-8a09-de7681eb8476 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:32.375758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e11d20b3-d024-4417-8a09-de7681eb8476 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:32.377926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 777/1531] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:16:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:32.383093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0758ae8-ee7b-4f34-891f-f2807c8d929f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:32.383547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0758ae8-ee7b-4f34-891f-f2807c8d929f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:32.383689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0758ae8-ee7b-4f34-891f-f2807c8d929f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:32.402091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d0758ae8-ee7b-4f34-891f-f2807c8d929f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:32.403396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d0758ae8-ee7b-4f34-891f-f2807c8d929f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:32.412263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 755/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:16:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:32.451770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d20b3-d024-4417-8a09-de7681eb8476 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:32.463085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e11d20b3-d024-4417-8a09-de7681eb8476 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:32.466976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25ac8cb-34ea-4ab0-adf2-6b3ba6b41617 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:32.467611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ac8cb-34ea-4ab0-adf2-6b3ba6b41617 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:32.467892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ac8cb-34ea-4ab0-adf2-6b3ba6b41617 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:32.468166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25ac8cb-34ea-4ab0-adf2-6b3ba6b41617 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:32.478165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ac8cb-34ea-4ab0-adf2-6b3ba6b41617 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:32.479635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f25ac8cb-34ea-4ab0-adf2-6b3ba6b41617 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:32.489878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 778/1533] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:35.990328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796b4e49-eefa-4f2f-981f-bb838fa9391e None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:35.990888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796b4e49-eefa-4f2f-981f-bb838fa9391e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:35.990888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-796b4e49-eefa-4f2f-981f-bb838fa9391e None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:35.992178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 756/1534] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:16:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:35.997171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424d8eb9-9346-4528-baba-bbfd9a1bb671 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:35.997579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424d8eb9-9346-4528-baba-bbfd9a1bb671 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:35.997730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424d8eb9-9346-4528-baba-bbfd9a1bb671 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:36.004847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:16:36.016510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-424d8eb9-9346-4528-baba-bbfd9a1bb671 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:36.019131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-424d8eb9-9346-4528-baba-bbfd9a1bb671 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:36.025925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 779/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:16:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:36.331148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796b4e49-eefa-4f2f-981f-bb838fa9391e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:36.332843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-796b4e49-eefa-4f2f-981f-bb838fa9391e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:36.410275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-796b4e49-eefa-4f2f-981f-bb838fa9391e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:36.413802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a401b985-5ace-49fd-aef0-b9306331a944 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:36.414691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401b985-5ace-49fd-aef0-b9306331a944 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:36.415004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401b985-5ace-49fd-aef0-b9306331a944 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:36.415281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a401b985-5ace-49fd-aef0-b9306331a944 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:36.424872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a401b985-5ace-49fd-aef0-b9306331a944 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:36.426289 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a401b985-5ace-49fd-aef0-b9306331a944 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:36.435770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 757/1536] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:38.474462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f805e4e-de64-4dfe-8706-0374eec28f5e None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:38.475082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f805e4e-de64-4dfe-8706-0374eec28f5e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:38.475082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f805e4e-de64-4dfe-8706-0374eec28f5e None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:38.476445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 780/1537] 127.0.0.1 () {54 vars in 916 bytes} [Thu Jul 14 12:16:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:16:38.481472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8341143e-dfed-4ff8-893a-595ce19f9742 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:38.481864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8341143e-dfed-4ff8-893a-595ce19f9742 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:38.482031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8341143e-dfed-4ff8-893a-595ce19f9742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:38.500472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-8341143e-dfed-4ff8-893a-595ce19f9742 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:16:38.502278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8341143e-dfed-4ff8-893a-595ce19f9742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:38.510102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 758/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Thu Jul 14 12:16:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:16:38.809363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f805e4e-de64-4dfe-8706-0374eec28f5e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:16:38.811100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7f805e4e-de64-4dfe-8706-0374eec28f5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:16:38.824569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f805e4e-de64-4dfe-8706-0374eec28f5e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:16:38.828138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44321bcb-39f3-4727-a7ed-c867f36318f8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:16:38.828793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44321bcb-39f3-4727-a7ed-c867f36318f8 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:16:38.829077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44321bcb-39f3-4727-a7ed-c867f36318f8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:16:38.829355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44321bcb-39f3-4727-a7ed-c867f36318f8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:16:38.838301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44321bcb-39f3-4727-a7ed-c867f36318f8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:16:38.839767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44321bcb-39f3-4727-a7ed-c867f36318f8 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:16:38.849212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 781/1539] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:16:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:16:48.495282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:16:48.845011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=8 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:11.805079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457f359f-210b-49b1-9204-52d8fe82f85b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:11.805729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457f359f-210b-49b1-9204-52d8fe82f85b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:11.805729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-457f359f-210b-49b1-9204-52d8fe82f85b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:11.832886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-457f359f-210b-49b1-9204-52d8fe82f85b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:11.834039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-457f359f-210b-49b1-9204-52d8fe82f85b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:11.843054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 759/1540] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:11.852568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab138b4-b666-40bb-806b-c9bb9e3ba1da None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:11.853615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab138b4-b666-40bb-806b-c9bb9e3ba1da None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:11.854227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab138b4-b666-40bb-806b-c9bb9e3ba1da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:11.883866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1ab138b4-b666-40bb-806b-c9bb9e3ba1da None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:11.885495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1ab138b4-b666-40bb-806b-c9bb9e3ba1da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:11.896758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 782/1541] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:11.901957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457f359f-210b-49b1-9204-52d8fe82f85b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:11.912962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-457f359f-210b-49b1-9204-52d8fe82f85b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:11.927517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:11.929139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:11.929139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:11.929139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:11.929422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:11.929730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:11.930754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING py.warnings [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:18:11.930754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: warnings.warn(msg) Jul 14 12:18:11.930754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: Jul 14 12:18:11.931870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:11.949128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 760/1542] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:11.957794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab138b4-b666-40bb-806b-c9bb9e3ba1da None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:11.968720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab138b4-b666-40bb-806b-c9bb9e3ba1da None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:11.984232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3373d2-df1f-4102-9aa4-c4e0bb181932 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:11.985324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3373d2-df1f-4102-9aa4-c4e0bb181932 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:11.985852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3373d2-df1f-4102-9aa4-c4e0bb181932 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:11.986356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3373d2-df1f-4102-9aa4-c4e0bb181932 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:11.987325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3373d2-df1f-4102-9aa4-c4e0bb181932 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:11.988591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-0f3373d2-df1f-4102-9aa4-c4e0bb181932 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:18:11.988591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:18:11.988591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:18:11.990537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3373d2-df1f-4102-9aa4-c4e0bb181932 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.149501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 783/1543] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:11 2022] POST /identity/v3/projects => generated 377 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:12.155080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.166026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e446fe7-adbc-47e3-a8ab-9014657a9a0f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.185034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a5bdc-206d-40c6-9f11-c3d05df2971f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.187498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a5bdc-206d-40c6-9f11-c3d05df2971f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.188253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e7a5bdc-206d-40c6-9f11-c3d05df2971f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.226791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1e7a5bdc-206d-40c6-9f11-c3d05df2971f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.228629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1e7a5bdc-206d-40c6-9f11-c3d05df2971f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.248186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.248869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.249147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.250518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.251266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.252301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING py.warnings [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:18:12.252301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: warnings.warn(msg) Jul 14 12:18:12.252301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: Jul 14 12:18:12.253522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.354756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 761/1544] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/users => generated 289 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:12.360158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.373798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 784/1545] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:12.378343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a1a5bfa-2fed-4eb5-8800-ac5e144a5933 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.384894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d3018-96a8-4593-84f5-255242032a8a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.385173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d3018-96a8-4593-84f5-255242032a8a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.385451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48d3018-96a8-4593-84f5-255242032a8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.393146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3525249-e627-4347-9383-57db229ac808 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.393829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3525249-e627-4347-9383-57db229ac808 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.394114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3525249-e627-4347-9383-57db229ac808 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.394389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3525249-e627-4347-9383-57db229ac808 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.395152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3525249-e627-4347-9383-57db229ac808 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.398399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3525249-e627-4347-9383-57db229ac808 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.423437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b48d3018-96a8-4593-84f5-255242032a8a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.424866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b48d3018-96a8-4593-84f5-255242032a8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.430941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 762/1546] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:12 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:12.432759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf98356-975a-4c4c-843c-e2575dab4b8f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.433279 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf98356-975a-4c4c-843c-e2575dab4b8f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.434060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf98356-975a-4c4c-843c-e2575dab4b8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.445259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 785/1547] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 61 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.446563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d3018-96a8-4593-84f5-255242032a8a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.463091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ddf98356-975a-4c4c-843c-e2575dab4b8f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.464742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ddf98356-975a-4c4c-843c-e2575dab4b8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.465722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48d3018-96a8-4593-84f5-255242032a8a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.477119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc8f1d14-ab3f-4cd2-a73c-bd14ba64b006 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.478927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f1d14-ab3f-4cd2-a73c-bd14ba64b006 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.481342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f1d14-ab3f-4cd2-a73c-bd14ba64b006 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.482109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc8f1d14-ab3f-4cd2-a73c-bd14ba64b006 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.483145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8f1d14-ab3f-4cd2-a73c-bd14ba64b006 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.483663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 763/1548] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.484759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf98356-975a-4c4c-843c-e2575dab4b8f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.487438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc8f1d14-ab3f-4cd2-a73c-bd14ba64b006 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.507181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf98356-975a-4c4c-843c-e2575dab4b8f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.509627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 786/1549] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:12 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:12.511185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176af40-56f1-42da-a10b-7ffcb9f193f9 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.511709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176af40-56f1-42da-a10b-7ffcb9f193f9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.512206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176af40-56f1-42da-a10b-7ffcb9f193f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.525048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.527449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.527965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.530059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.531067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:12.531782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.534670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.565571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b176af40-56f1-42da-a10b-7ffcb9f193f9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.569165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b176af40-56f1-42da-a10b-7ffcb9f193f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.571655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 764/1550] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:12.575255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.598939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 787/1551] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.602157 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4be5b-2935-483b-85f1-b68f31e62c08 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.604891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4be5b-2935-483b-85f1-b68f31e62c08 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.605663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b4be5b-2935-483b-85f1-b68f31e62c08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.611026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c30b85f-14a8-4149-a0fb-bb49362277ec admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.620226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.627092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.627422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.627710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] PATH_INFO: `/v3/projects/33b09a1d039248aa95a089c8d8d99c68/users/d6da4b0616034aa1bb22c44a8f5ef280/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.641865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-29b4be5b-2935-483b-85f1-b68f31e62c08 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.643138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-29b4be5b-2935-483b-85f1-b68f31e62c08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.658154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] RBAC: Authorizing `identity:create_grant(project_id=33b09a1d039248aa95a089c8d8d99c68, user_id=d6da4b0616034aa1bb22c44a8f5ef280, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.661104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.664858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 788/1552] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.665698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b4be5b-2935-483b-85f1-b68f31e62c08 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.684072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b4be5b-2935-483b-85f1-b68f31e62c08 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.687172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e157c3db-cac0-43a1-915b-9a7899b375cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.687841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e157c3db-cac0-43a1-915b-9a7899b375cd admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.688129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e157c3db-cac0-43a1-915b-9a7899b375cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.688402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e157c3db-cac0-43a1-915b-9a7899b375cd admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.689087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e157c3db-cac0-43a1-915b-9a7899b375cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.691095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e157c3db-cac0-43a1-915b-9a7899b375cd admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.732495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 765/1553] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:12 2022] PUT /identity/v3/projects/33b09a1d039248aa95a089c8d8d99c68/users/d6da4b0616034aa1bb22c44a8f5ef280/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:12.744419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.805679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8334ea-b9ee-44c6-b3f1-538d4e3e845c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.810670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e4c16b-8131-4498-8d96-3ca70175a295 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.811385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e4c16b-8131-4498-8d96-3ca70175a295 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.811672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e4c16b-8131-4498-8d96-3ca70175a295 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.811950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e4c16b-8131-4498-8d96-3ca70175a295 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.812681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e4c16b-8131-4498-8d96-3ca70175a295 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:12.812995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e4c16b-8131-4498-8d96-3ca70175a295 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.814953 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e4c16b-8131-4498-8d96-3ca70175a295 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:12.827036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 766/1554] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:12.828468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1039df6-921d-4483-a22f-81c180f0a4b1 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.828696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1039df6-921d-4483-a22f-81c180f0a4b1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.828973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1039df6-921d-4483-a22f-81c180f0a4b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.839897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 789/1555] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/projects => generated 387 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:12.839897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357934d8-6824-4f1d-885d-2d85b1d36272 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.839897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357934d8-6824-4f1d-885d-2d85b1d36272 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.839897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357934d8-6824-4f1d-885d-2d85b1d36272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.847497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e1039df6-921d-4483-a22f-81c180f0a4b1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.848709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e1039df6-921d-4483-a22f-81c180f0a4b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.853077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:12.859101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-357934d8-6824-4f1d-885d-2d85b1d36272 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.860320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-357934d8-6824-4f1d-885d-2d85b1d36272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.912178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 767/1556] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.913498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a8b227-9bb9-43df-ad70-437fa775dbe1 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.913786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a8b227-9bb9-43df-ad70-437fa775dbe1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.914051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80a8b227-9bb9-43df-ad70-437fa775dbe1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.924167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 790/1557] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.925467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651ee547-612b-4fd9-8e04-add8dd9014b6 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.925915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651ee547-612b-4fd9-8e04-add8dd9014b6 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.926201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651ee547-612b-4fd9-8e04-add8dd9014b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.932529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-80a8b227-9bb9-43df-ad70-437fa775dbe1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.933745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-80a8b227-9bb9-43df-ad70-437fa775dbe1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.942565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 768/1558] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.943594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce4bcf-f9b6-4d8b-bdf6-2434f5a56ccf None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.943838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce4bcf-f9b6-4d8b-bdf6-2434f5a56ccf None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.944115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fce4bcf-f9b6-4d8b-bdf6-2434f5a56ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.950076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-651ee547-612b-4fd9-8e04-add8dd9014b6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.951533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-651ee547-612b-4fd9-8e04-add8dd9014b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.959709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 791/1559] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.960929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d0714c-5101-4736-b0d6-1e450c3f73bd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.961219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d0714c-5101-4736-b0d6-1e450c3f73bd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.961493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d0714c-5101-4736-b0d6-1e450c3f73bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.962606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-5fce4bcf-f9b6-4d8b-bdf6-2434f5a56ccf None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.963981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5fce4bcf-f9b6-4d8b-bdf6-2434f5a56ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.969136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:12.972965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 769/1560] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.973739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fce4bcf-f9b6-4d8b-bdf6-2434f5a56ccf None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.980141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-80d0714c-5101-4736-b0d6-1e450c3f73bd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:12.981440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-80d0714c-5101-4736-b0d6-1e450c3f73bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:12.984578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fce4bcf-f9b6-4d8b-bdf6-2434f5a56ccf None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:12.994309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82156d25-8b9d-4bdf-acc2-beaf2b0fcda0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:12.994309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82156d25-8b9d-4bdf-acc2-beaf2b0fcda0 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:12.994551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 792/1561] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:12.994656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82156d25-8b9d-4bdf-acc2-beaf2b0fcda0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:12.994656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82156d25-8b9d-4bdf-acc2-beaf2b0fcda0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:12.994880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d0714c-5101-4736-b0d6-1e450c3f73bd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:12.994975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82156d25-8b9d-4bdf-acc2-beaf2b0fcda0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:12.994975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82156d25-8b9d-4bdf-acc2-beaf2b0fcda0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.005689 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d0714c-5101-4736-b0d6-1e450c3f73bd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.010379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.011078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.011363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.011747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.012406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.014359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.118122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 770/1562] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/projects => generated 383 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.118122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0be7ee-5412-4482-8740-d7595ef1db07 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.118122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0be7ee-5412-4482-8740-d7595ef1db07 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.118122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0be7ee-5412-4482-8740-d7595ef1db07 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.135406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-cf0be7ee-5412-4482-8740-d7595ef1db07 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:13.137445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cf0be7ee-5412-4482-8740-d7595ef1db07 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:13.144219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 793/1563] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:12 2022] POST /identity/v3/users => generated 294 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.145236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.155359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da44d10c-6fbe-4158-af3e-cc27fda78e42 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.199944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 771/1564] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:13.201749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62304cc7-7e73-4a3e-8e45-14739e7789e9 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.202038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62304cc7-7e73-4a3e-8e45-14739e7789e9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.202319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62304cc7-7e73-4a3e-8e45-14739e7789e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.217962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.218634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.218949 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.219227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.220011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:13.220419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.221238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-62304cc7-7e73-4a3e-8e45-14739e7789e9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:13.222115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.222460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-62304cc7-7e73-4a3e-8e45-14739e7789e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:13.230340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 772/1565] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:13.234264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 794/1566] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:13.235110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.236283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e684b-03bb-42b6-a20c-d946a4673261 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.236583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e684b-03bb-42b6-a20c-d946a4673261 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.236772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:13.236872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e684b-03bb-42b6-a20c-d946a4673261 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.254825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f04e684b-03bb-42b6-a20c-d946a4673261 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:13.256272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f04e684b-03bb-42b6-a20c-d946a4673261 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:13.265129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 773/1567] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:13.266265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d6606-8f58-4f35-912c-159201123263 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.266553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d6606-8f58-4f35-912c-159201123263 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.266870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d6606-8f58-4f35-912c-159201123263 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.286272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7f9d6606-8f58-4f35-912c-159201123263 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:13.287496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7f9d6606-8f58-4f35-912c-159201123263 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:13.295678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 774/1568] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:13.298349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb33916-45b9-4883-a7dd-ce078bf8a3cb admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.299668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9d6606-8f58-4f35-912c-159201123263 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.302244 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.302924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.303211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.303485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.308957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9d6606-8f58-4f35-912c-159201123263 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.312709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.314161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.314524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.315205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.315502 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.315758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.316445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.318350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.323880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 795/1569] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:13.324532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.334663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d11808-f58b-481e-8e78-1c89aec98d56 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.339490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.344089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.344089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.344089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.344089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.344089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.354159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 796/1570] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:13.355201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.364546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b9aeff-afed-479f-b938-9564e3b1732c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.369161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.369911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.370194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.370472 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.371203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:13.371505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.373653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.386150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 797/1571] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:13.387093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.396779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0923a77b-132e-4335-9c1d-f8817abdd0e7 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.401627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.402371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.402657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.403012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.409631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:13.409631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.409631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.409891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 775/1572] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/users => generated 292 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.409891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.416782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97618826-babd-4621-a194-62b9c69cab05 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.419326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 798/1573] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:13.424420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.424652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.424652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.424652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.424652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.425059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.428402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.436564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ca161c-65bb-4034-9760-b6c1dc03a019 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.441401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.442124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.442420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.442699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.443462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:13.443780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.445383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.459369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 799/1574] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:13.460121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.470658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aafad81f-d637-4da5-915c-369498267c58 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.475590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.476248 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.476564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.476841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.477568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:13.478012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.479541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.491573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 800/1575] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:13.492536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.503717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef439e0-dec0-4a13-8041-01a6eb1cea47 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.510439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 776/1576] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/projects => generated 393 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.510439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.512399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.513907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.514204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.514487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.515429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.516880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.526298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:13.528801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 801/1577] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:13.529610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.535894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:13.540745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de77d684-311e-4bf9-b8ec-4f91da905ece admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.571107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-056686d3-03e0-45fc-b6d6-c365432ce6dd admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.576381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.577158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.577448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.577727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.578573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.580538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.598731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.608651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.608651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.608651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.608651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.608651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.612754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 802/1578] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:13 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:13.613553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.626386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6124320-e0f2-40f5-9550-3eb656320613 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.631270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.632000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.632363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.632648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.633344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.635317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.696596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 777/1579] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/projects => generated 373 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.696596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.724587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 803/1580] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/projects => generated 365 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.725684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.762960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510170e5-d9db-4a6f-be38-de109aa359dd admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.787967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491dcae9-7ad0-48c7-90f4-8b09059963b9 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:13.792825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.793477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.793755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.794028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.795283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.797160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.822171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:13.822863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:13.823146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:13.823424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:13.824112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:13.826181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:13.932224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 804/1581] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/projects => generated 373 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.933189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:13.994081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 778/1582] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/projects => generated 373 bytes in 297 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:13.994081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.006384 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c3ebc1-399b-4d03-bda9-3706d14f5b68 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.011831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:14.031359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:14.064508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce4679f-0b27-455b-8920-75eb17652c5c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.069312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.070206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.070488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.070955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] PATH_INFO: `/v3/projects/8b9dfc22855a4c83883bd69d52ffdac1/users/a5b84dec5a8d4a47bb4712d03819affc/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.071543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.072283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.072627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.072915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.073823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.076271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.086678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b9dfc22855a4c83883bd69d52ffdac1, user_id=a5b84dec5a8d4a47bb4712d03819affc, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.089622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.181437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 779/1584] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:13 2022] PUT /identity/v3/projects/8b9dfc22855a4c83883bd69d52ffdac1/users/a5b84dec5a8d4a47bb4712d03819affc/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:14.181653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 805/1584] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:13 2022] POST /identity/v3/users => generated 297 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:14.181653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.184320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.200271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a13404-ab25-4f64-a0d8-fda5219d0d42 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.258916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e2ca378-4ca6-43a4-b980-69717587251e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.265344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.266115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.266468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.266845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.267397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.267712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.268079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.268366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.268645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.269343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.270314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.271482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.285491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 780/1585] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.285491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.296240 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ea12ce-115f-4e69-a4e0-1f865162caf9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.300783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.312294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.312294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.312294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.312294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.312294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.365267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 806/1586] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:14 2022] POST /identity/v3/users => generated 287 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:14.366073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.376032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128d2637-c35e-419b-8d2e-ceb3404881e2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.380634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.381360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.381644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.381921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.382601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.384555 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.396321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 781/1587] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:14 2022] POST /identity/v3/users => generated 283 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:14.398957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.407790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cff95e4-437b-4f6b-a38e-60cbf6aed3f9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.412285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.412937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.413215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.413486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.414191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.416129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.490704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 807/1588] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:14 2022] POST /identity/v3/users => generated 287 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:14.491562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.500868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b848fcd-03af-4124-91ef-dece5947dd6c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.505402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db716ab5-6115-4860-8fc7-0c1f693ba1bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.506053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db716ab5-6115-4860-8fc7-0c1f693ba1bc admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.506337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db716ab5-6115-4860-8fc7-0c1f693ba1bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.506611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db716ab5-6115-4860-8fc7-0c1f693ba1bc admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.507355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db716ab5-6115-4860-8fc7-0c1f693ba1bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.508981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db716ab5-6115-4860-8fc7-0c1f693ba1bc admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.519482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 808/1589] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.520627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5034e-51b6-469b-a807-b51e2c6fbd65 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.520914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5034e-51b6-469b-a807-b51e2c6fbd65 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.521190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa5034e-51b6-469b-a807-b51e2c6fbd65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.547690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-faa5034e-51b6-469b-a807-b51e2c6fbd65 None None] MFA Rules not processed for user `a5b84dec5a8d4a47bb4712d03819affc`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:14.548890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-faa5034e-51b6-469b-a807-b51e2c6fbd65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:14.574206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 782/1590] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:14 2022] POST /identity/v3/users => generated 287 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:14.575175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.585979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d727f44-08b2-4c04-a16d-949b7b008cca admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.592654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.593611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.593895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.594173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] PATH_INFO: `/v3/projects/e8c82e1c7ee74798b03deba105d1cbf0/users/09f5c519c42f484b990b89ad5d3c93d5/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.599594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=91,keystone:INSERT=9 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:14.610631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e8c82e1c7ee74798b03deba105d1cbf0, user_id=09f5c519c42f484b990b89ad5d3c93d5, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.613421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.618731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 809/1591] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:14.623040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa5034e-51b6-469b-a807-b51e2c6fbd65 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.633271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa5034e-51b6-469b-a807-b51e2c6fbd65 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.639082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.639762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.640055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.642349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.642349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.643976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.660920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 810/1592] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.665034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.675128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9cdc75d-a538-4179-8b98-6bcb560f4b1d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.679924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.680586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.680871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.681144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.681869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.683536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.695773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 811/1593] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.696411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.708904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23898145-447c-4f1b-819f-8072b6f41115 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.709157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 783/1594] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:14 2022] PUT /identity/v3/projects/e8c82e1c7ee74798b03deba105d1cbf0/users/09f5c519c42f484b990b89ad5d3c93d5/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:14.710094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.771802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.772395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.772676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.772951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.773672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.775308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.781933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:14.786208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 812/1595] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.787125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.793109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9647329b-1cf4-494d-8065-3ec18305f601 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.797401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45f744b6-e15e-41ef-a2e5-dc8470634242 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.798209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.798910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.799197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.799477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.800241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.801873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.802131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.802794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.803080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.803355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.804065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.805645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.815628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 784/1596] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.815628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.816633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 813/1597] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.817353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.823653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c4ea4e-22a4-4d81-a6d7-e899e5a84491 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.828156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97514bb1-9d1c-4859-b88f-56934c3c1984 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.829413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.830123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.830409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.830729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.831512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.833483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.833602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.834136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.834421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.834699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.835454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.837038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.848300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 814/1598] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.849088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.858500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9475da57-ecf4-43b3-94b6-8124f42d1115 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.863095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3670cbb7-71d1-4c28-a072-00520e28025d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.863729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3670cbb7-71d1-4c28-a072-00520e28025d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.864011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3670cbb7-71d1-4c28-a072-00520e28025d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.864287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3670cbb7-71d1-4c28-a072-00520e28025d admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.864994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3670cbb7-71d1-4c28-a072-00520e28025d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.866635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3670cbb7-71d1-4c28-a072-00520e28025d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:14.876494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 815/1599] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:14.877726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2572c379-1312-42cf-a54f-437cf8f45fdd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.878011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2572c379-1312-42cf-a54f-437cf8f45fdd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.878288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2572c379-1312-42cf-a54f-437cf8f45fdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.905084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2572c379-1312-42cf-a54f-437cf8f45fdd None None] MFA Rules not processed for user `09f5c519c42f484b990b89ad5d3c93d5`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:14.906271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2572c379-1312-42cf-a54f-437cf8f45fdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:14.972734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 816/1600] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:14.973596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2572c379-1312-42cf-a54f-437cf8f45fdd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:14.986690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2572c379-1312-42cf-a54f-437cf8f45fdd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:14.991362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:14.992122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:14.992411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:14.992689 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:14.993423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:14.995075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.006252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 817/1601] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:14 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:15.007101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:15.021729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712dedc7-1aeb-45c3-a527-26013527f344 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:15.024249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 785/1602] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:14 2022] POST /identity/v3/projects => generated 389 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:15.025025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:15.033721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:15.033721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.033721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.033721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] PATH_INFO: `/v3/projects/bb62d84584b3426687390833270648d6/users/0d302e57946c42d799e9bf120873eba6/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.036990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:15.051115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=bb62d84584b3426687390833270648d6, user_id=0d302e57946c42d799e9bf120873eba6, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:15.052358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:18:15.052358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:18:15.052358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:18:15.054430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.099651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3e67a1-c4cf-45ad-a080-a3c35b8478ad admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:15.104738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:15.105497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.105781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.106059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.106863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:15.108542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.119331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 786/1603] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:15 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:15.120158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:15.130842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd37e3c-9253-467a-b41b-9731b5f3751a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:15.135533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e113f646-4fe3-4cf8-ae0c-04e8ec43fbca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:15.136428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113f646-4fe3-4cf8-ae0c-04e8ec43fbca admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.136829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113f646-4fe3-4cf8-ae0c-04e8ec43fbca admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.137067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e113f646-4fe3-4cf8-ae0c-04e8ec43fbca admin admin] PATH_INFO: `/v3/projects/1500083e98d84701b246fbbd749354fa/users/a0a6364a4a884b09b5eb80897c19e296/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.155022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e113f646-4fe3-4cf8-ae0c-04e8ec43fbca admin admin] RBAC: Authorizing `identity:create_grant(project_id=1500083e98d84701b246fbbd749354fa, user_id=a0a6364a4a884b09b5eb80897c19e296, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:15.157773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e113f646-4fe3-4cf8-ae0c-04e8ec43fbca admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.223182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 818/1604] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:15 2022] PUT /identity/v3/projects/bb62d84584b3426687390833270648d6/users/0d302e57946c42d799e9bf120873eba6/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:15.223786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:15.286610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 787/1605] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:15 2022] PUT /identity/v3/projects/1500083e98d84701b246fbbd749354fa/users/a0a6364a4a884b09b5eb80897c19e296/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:15.288052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4e401e-63f9-46a5-8b38-fa55ed2578f1 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.288347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4e401e-63f9-46a5-8b38-fa55ed2578f1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.288739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4e401e-63f9-46a5-8b38-fa55ed2578f1 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.290325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 788/1606] 10.222.0.27 () {54 vars in 961 bytes} [Thu Jul 14 12:18:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:15.291117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4e401e-63f9-46a5-8b38-fa55ed2578f1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:15.300080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab22a22-37a4-4058-b880-70e600eb5e3e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:15.375361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb4e401e-63f9-46a5-8b38-fa55ed2578f1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:15.380164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:15.381467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.381598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.381828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] PATH_INFO: `/v3/projects/452717ce7f314047b3664d11fd81be3a/users/361a4f34e0b0435e944b6fe1ab485a1a/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.382357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:15.383468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.383760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.384333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] PATH_INFO: `/v3/projects/78c0c7328a8b4727812b1222e4d5e430/users/414d54ec8e2e466596ce921a8dadde3a/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.400245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] RBAC: Authorizing `identity:create_grant(project_id=452717ce7f314047b3664d11fd81be3a, user_id=361a4f34e0b0435e944b6fe1ab485a1a, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:15.402794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=78c0c7328a8b4727812b1222e4d5e430, user_id=414d54ec8e2e466596ce921a8dadde3a, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:15.403010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.407174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.521255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 789/1607] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:15 2022] PUT /identity/v3/projects/452717ce7f314047b3664d11fd81be3a/users/361a4f34e0b0435e944b6fe1ab485a1a/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:15.522420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:15.523345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 819/1608] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:15 2022] PUT /identity/v3/projects/78c0c7328a8b4727812b1222e4d5e430/users/414d54ec8e2e466596ce921a8dadde3a/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:15.524899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:15.552948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:15.620786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df9a3eb9-2481-459b-a5e9-7c5bc1d8f833 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:15.626105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03256bed-19f9-4a13-8fd7-1c4acecf71fc admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:15.629680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07371da5-e7b7-4b1d-82fb-5e5587603ea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:15.630751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07371da5-e7b7-4b1d-82fb-5e5587603ea8 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.631221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07371da5-e7b7-4b1d-82fb-5e5587603ea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.631605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6bee9ac-bf2b-4ca6-a3f1-826c2e79907d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:15.631735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07371da5-e7b7-4b1d-82fb-5e5587603ea8 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.632699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07371da5-e7b7-4b1d-82fb-5e5587603ea8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:15.632826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bee9ac-bf2b-4ca6-a3f1-826c2e79907d admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.633053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bee9ac-bf2b-4ca6-a3f1-826c2e79907d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.633338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6bee9ac-bf2b-4ca6-a3f1-826c2e79907d admin admin] PATH_INFO: `/v3/projects/e24bf8b5dd8247039f6a5ac09db96698/users/89f09a31bcaa4553b210a9222c3a794d/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.635116 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07371da5-e7b7-4b1d-82fb-5e5587603ea8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.651215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bee9ac-bf2b-4ca6-a3f1-826c2e79907d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e24bf8b5dd8247039f6a5ac09db96698, user_id=89f09a31bcaa4553b210a9222c3a794d, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:15.655194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6bee9ac-bf2b-4ca6-a3f1-826c2e79907d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:15.766293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 790/1609] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:15 2022] POST /identity/v3/users => generated 301 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:15.767736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18e64f6-e36d-408d-aa83-bef78594cc8c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.768030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18e64f6-e36d-408d-aa83-bef78594cc8c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.768315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18e64f6-e36d-408d-aa83-bef78594cc8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.785902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-a18e64f6-e36d-408d-aa83-bef78594cc8c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:15.787121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a18e64f6-e36d-408d-aa83-bef78594cc8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:15.795796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:15.799864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 791/1610] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:15.799864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11004dcb-11d3-46e3-8b28-93c2f8088c4a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.799864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11004dcb-11d3-46e3-8b28-93c2f8088c4a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.799864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11004dcb-11d3-46e3-8b28-93c2f8088c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.805687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 820/1611] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:15 2022] PUT /identity/v3/projects/e24bf8b5dd8247039f6a5ac09db96698/users/89f09a31bcaa4553b210a9222c3a794d/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:15.807443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d268485e-dad1-4175-8faa-adcb9a9e0478 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.808282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d268485e-dad1-4175-8faa-adcb9a9e0478 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.808565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d268485e-dad1-4175-8faa-adcb9a9e0478 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.827744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-11004dcb-11d3-46e3-8b28-93c2f8088c4a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:15.829215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-11004dcb-11d3-46e3-8b28-93c2f8088c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:15.849970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d268485e-dad1-4175-8faa-adcb9a9e0478 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:15.851470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d268485e-dad1-4175-8faa-adcb9a9e0478 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:15.898831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 792/1612] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:15.898831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325fadd-2706-4033-a081-ffa4a597ac99 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.898831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325fadd-2706-4033-a081-ffa4a597ac99 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.898831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1325fadd-2706-4033-a081-ffa4a597ac99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.936810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 821/1613] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Jul 14 12:18:15 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:15.939246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a9924-514b-4940-b8c3-1986f7c04542 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:15.939246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a9924-514b-4940-b8c3-1986f7c04542 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:15.939524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22a9924-514b-4940-b8c3-1986f7c04542 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:15.947594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1325fadd-2706-4033-a081-ffa4a597ac99 None None] MFA Rules not processed for user `361a4f34e0b0435e944b6fe1ab485a1a`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:15.949446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1325fadd-2706-4033-a081-ffa4a597ac99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:15.999306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a22a9924-514b-4940-b8c3-1986f7c04542 None None] MFA Rules not processed for user `414d54ec8e2e466596ce921a8dadde3a`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:16.000547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a22a9924-514b-4940-b8c3-1986f7c04542 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:16.049995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 793/1614] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:15 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:16.051153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325fadd-2706-4033-a081-ffa4a597ac99 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.067752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1325fadd-2706-4033-a081-ffa4a597ac99 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.073666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.074350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.074634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.074994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.076397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.078168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.096101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 794/1615] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:16 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:16.096101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.096408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 822/1616] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:15 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:16.096408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cecd73-3c89-45fb-9931-f6a0ad14cc04 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.096408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cecd73-3c89-45fb-9931-f6a0ad14cc04 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.096408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19cecd73-3c89-45fb-9931-f6a0ad14cc04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.130261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-19cecd73-3c89-45fb-9931-f6a0ad14cc04 None None] MFA Rules not processed for user `89f09a31bcaa4553b210a9222c3a794d`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:16.131629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-19cecd73-3c89-45fb-9931-f6a0ad14cc04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:16.136040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:16.178943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-818a11ae-8e6e-4c58-8904-19b85f8619f0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.183708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.184581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.186973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.188034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.201866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.206173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.326697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 795/1617] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:16.326697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.328121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:16.328415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb462573-4804-46aa-ba9b-abc0b3cdb958 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.328415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.328415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.328415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.328746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.328830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 823/1618] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:16 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:16.328830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cecd73-3c89-45fb-9931-f6a0ad14cc04 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.328977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.328977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.329487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19cecd73-3c89-45fb-9931-f6a0ad14cc04 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.329565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 796/1619] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:16.329565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.329786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.329786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.329786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.329786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.330074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a329ef52-93fe-4fe9-bb24-494532dc1778 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.344962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.346022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.346369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.346850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.347715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.357787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.400589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.402338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.413524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 824/1620] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:16 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:16.414418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.428288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b57728ae-b7e9-4b50-9365-310446a243c9 service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.433130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b5bfec9-a07a-4430-a31d-c567e73797a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.436027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5bfec9-a07a-4430-a31d-c567e73797a4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.436027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5bfec9-a07a-4430-a31d-c567e73797a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.436027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b5bfec9-a07a-4430-a31d-c567e73797a4 admin admin] PATH_INFO: `/v3/projects/b745718edc06441e9c0dfbcd98673e23/users/929c363a7df742748245b912602a7ed2/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.453280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5bfec9-a07a-4430-a31d-c567e73797a4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b745718edc06441e9c0dfbcd98673e23, user_id=929c363a7df742748245b912602a7ed2, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.471047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 797/1621] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:16 2022] POST /identity/v3/projects => generated 373 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:16.471986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.472565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b5bfec9-a07a-4430-a31d-c567e73797a4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.558650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9504154-43c5-413a-b72f-2ade4fe124c1 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.578737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.584767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 825/1622] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:16 2022] PUT /identity/v3/projects/b745718edc06441e9c0dfbcd98673e23/users/929c363a7df742748245b912602a7ed2/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:16.586474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417c66ad-0e0e-4d88-8925-93b1aabbfde7 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.586907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417c66ad-0e0e-4d88-8925-93b1aabbfde7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.587359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.587693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.587958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417c66ad-0e0e-4d88-8925-93b1aabbfde7 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.589173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.589911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.590496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 826/1623] 10.222.0.27 () {54 vars in 961 bytes} [Thu Jul 14 12:18:16 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:16.598547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417c66ad-0e0e-4d88-8925-93b1aabbfde7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.599913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.706338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417c66ad-0e0e-4d88-8925-93b1aabbfde7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.715378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a37393-4c51-4547-8fd7-64ac4cdcead7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.716120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a37393-4c51-4547-8fd7-64ac4cdcead7 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.716451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a37393-4c51-4547-8fd7-64ac4cdcead7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.716945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a37393-4c51-4547-8fd7-64ac4cdcead7 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.717788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a37393-4c51-4547-8fd7-64ac4cdcead7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.720067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a37393-4c51-4547-8fd7-64ac4cdcead7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.725612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 798/1624] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:16 2022] POST /identity/v3/projects => generated 363 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:16.729342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.799298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ecd719-a81b-47eb-a559-9991626dec68 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.803996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f1c7fb-0f07-4a8d-912b-ea5995e5c729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.804669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f1c7fb-0f07-4a8d-912b-ea5995e5c729 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.804950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f1c7fb-0f07-4a8d-912b-ea5995e5c729 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.805224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f1c7fb-0f07-4a8d-912b-ea5995e5c729 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.806278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f1c7fb-0f07-4a8d-912b-ea5995e5c729 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.807591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2f1c7fb-0f07-4a8d-912b-ea5995e5c729 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.819391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 799/1625] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:16 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:16.820865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e5addd-d4a3-42c5-a052-0e47df0cb320 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.821296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e5addd-d4a3-42c5-a052-0e47df0cb320 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.821579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e5addd-d4a3-42c5-a052-0e47df0cb320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.851303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 827/1626] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:16 2022] POST /identity/v3/users => generated 293 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:16.853502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a64efc-3f01-4d96-9d77-722eaca77d34 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.854279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a64efc-3f01-4d96-9d77-722eaca77d34 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.854819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a64efc-3f01-4d96-9d77-722eaca77d34 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.856086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-04e5addd-d4a3-42c5-a052-0e47df0cb320 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:16.857924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-04e5addd-d4a3-42c5-a052-0e47df0cb320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:16.864188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 828/1627] 10.222.0.27 () {54 vars in 912 bytes} [Thu Jul 14 12:18:16 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:16.866531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a64efc-3f01-4d96-9d77-722eaca77d34 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:16.885721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:16.891256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a64efc-3f01-4d96-9d77-722eaca77d34 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:16.908500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:16.908500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:16.908500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:16.908500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:16.908907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:16.908907 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:16.975036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 800/1628] 10.222.0.27 () {58 vars in 1058 bytes} [Thu Jul 14 12:18:16 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:16.977124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e5addd-d4a3-42c5-a052-0e47df0cb320 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.008517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e5addd-d4a3-42c5-a052-0e47df0cb320 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:17.016221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1ac7d0d-fe85-4049-a9d7-925bcf011ea8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:17.017474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ac7d0d-fe85-4049-a9d7-925bcf011ea8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.017912 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ac7d0d-fe85-4049-a9d7-925bcf011ea8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.018343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ac7d0d-fe85-4049-a9d7-925bcf011ea8 admin admin] PATH_INFO: `/v3/projects/b745718edc06441e9c0dfbcd98673e23/users/929c363a7df742748245b912602a7ed2/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.020797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ac7d0d-fe85-4049-a9d7-925bcf011ea8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b745718edc06441e9c0dfbcd98673e23, user_id=929c363a7df742748245b912602a7ed2, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:17.023969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1ac7d0d-fe85-4049-a9d7-925bcf011ea8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:17.040622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 829/1629] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:16 2022] POST /identity/v3/users => generated 288 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:17.045683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.057600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 801/1630] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:16 2022] PUT /identity/v3/projects/b745718edc06441e9c0dfbcd98673e23/users/929c363a7df742748245b912602a7ed2/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:17.061228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a0aeb1-226c-43df-b070-662e16615d0d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.062046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a0aeb1-226c-43df-b070-662e16615d0d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.062046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a0aeb1-226c-43df-b070-662e16615d0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.070416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-572a0499-2f6f-4988-881f-c3c35d9e948a admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:17.105820 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-72a0aeb1-226c-43df-b070-662e16615d0d None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:17.108958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-72a0aeb1-226c-43df-b070-662e16615d0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:17.159684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:17.181859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:17.187444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.187444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.187444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.187444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:17.187444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:17.208759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 830/1631] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:17 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:17.208759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.280380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 802/1632] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:18:17 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:17.282870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a0aeb1-226c-43df-b070-662e16615d0d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.316621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a0aeb1-226c-43df-b070-662e16615d0d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:17.327957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb79e74d-ef34-4f40-92e1-d0c6126bc6b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:17.328656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb79e74d-ef34-4f40-92e1-d0c6126bc6b3 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.330521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb79e74d-ef34-4f40-92e1-d0c6126bc6b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.330845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb79e74d-ef34-4f40-92e1-d0c6126bc6b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.332557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb79e74d-ef34-4f40-92e1-d0c6126bc6b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:17.334244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb79e74d-ef34-4f40-92e1-d0c6126bc6b3 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:17.354538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 803/1633] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:17 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:17.358994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565cfe42-7eb9-4bd9-8130-a451cdf51593 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.362711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565cfe42-7eb9-4bd9-8130-a451cdf51593 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.363031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565cfe42-7eb9-4bd9-8130-a451cdf51593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.378192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a58389e-6564-43b0-ad81-bbd214f08afd admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:17.392625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:17.400175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.400175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.400175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.402998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-565cfe42-7eb9-4bd9-8130-a451cdf51593 None None] MFA Rules not processed for user `929c363a7df742748245b912602a7ed2`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:17.404336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-565cfe42-7eb9-4bd9-8130-a451cdf51593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:17.479429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:17.481198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:17.493457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 831/1634] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:17 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:17.494374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.511434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aaefe7-7317-4652-829c-b5020878262e service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:17.524914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 804/1635] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:17 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:17.526053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565cfe42-7eb9-4bd9-8130-a451cdf51593 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.529176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:17.535806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.536141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.536651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] PATH_INFO: `/v3/projects/bbad5980e6ad45aba5b72ff0b01d5c6b/users/47e6a08ab9584b3abaf3becf7c39e0b9/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.573423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bbad5980e6ad45aba5b72ff0b01d5c6b, user_id=47e6a08ab9584b3abaf3becf7c39e0b9, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:17.578301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:17.656581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 832/1636] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:17 2022] PUT /identity/v3/projects/bbad5980e6ad45aba5b72ff0b01d5c6b/users/47e6a08ab9584b3abaf3becf7c39e0b9/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:17.657909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.681705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565cfe42-7eb9-4bd9-8130-a451cdf51593 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:17.795237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:17.795946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.796976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.797389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.816111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b161e26d-c849-4656-8063-4b4d0e8fa66a admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:17.828550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:17.831765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:17.831765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:17.831765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] PATH_INFO: `/v3/projects/9fbca917b8174b7c90db70744798b0ba/users/5986abaea4514de0b1b226ed76f08bb8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:17.872021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fbca917b8174b7c90db70744798b0ba, user_id=5986abaea4514de0b1b226ed76f08bb8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:17.874350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:17.959951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:17.962482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:17.977276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 805/1637] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:17 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 451 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:17.981143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:17.997159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e51460a-fe9e-43b4-a839-6b21bd4fdd49 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:18.004515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:18.005270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.005552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.005871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.020611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 833/1638] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:17 2022] PUT /identity/v3/projects/9fbca917b8174b7c90db70744798b0ba/users/5986abaea4514de0b1b226ed76f08bb8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:18.020611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:18.049194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:18.096813 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:18.130379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:18.135732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:18.144978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed693391-97b0-4539-a27a-790c024c675e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:18.156460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b97310c-b151-4c53-8d65-ad9b2c93bda1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:18.157222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97310c-b151-4c53-8d65-ad9b2c93bda1 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.157429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97310c-b151-4c53-8d65-ad9b2c93bda1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.157740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b97310c-b151-4c53-8d65-ad9b2c93bda1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.158613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97310c-b151-4c53-8d65-ad9b2c93bda1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:18.181242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 806/1639] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:17 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:18.182742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b97310c-b151-4c53-8d65-ad9b2c93bda1 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:18.183333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:18.239389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc441853-83a7-4a56-8c4b-59e68232fb48 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:18.239610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 834/1640] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:18 2022] GET /identity/v3/roles => generated 1420 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:18.239610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6117c89f-db86-4f22-9e87-514fc9001566 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.239610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6117c89f-db86-4f22-9e87-514fc9001566 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.239610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6117c89f-db86-4f22-9e87-514fc9001566 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.246586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 835/1641] 10.222.0.27 () {54 vars in 912 bytes} [Thu Jul 14 12:18:18 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:18.251303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6117c89f-db86-4f22-9e87-514fc9001566 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:18.251487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b5887c1-5aee-48ea-8e9b-cc716ebaef21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:18.254256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5887c1-5aee-48ea-8e9b-cc716ebaef21 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.256141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5887c1-5aee-48ea-8e9b-cc716ebaef21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.263132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b5887c1-5aee-48ea-8e9b-cc716ebaef21 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.263132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5887c1-5aee-48ea-8e9b-cc716ebaef21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:18.264558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b5887c1-5aee-48ea-8e9b-cc716ebaef21 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:18.272825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6117c89f-db86-4f22-9e87-514fc9001566 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:18.284317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 807/1642] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:18 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:18.286496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f41b4-db4a-456b-8219-22b8524c08e7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.286962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f41b4-db4a-456b-8219-22b8524c08e7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.292459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6f41b4-db4a-456b-8219-22b8524c08e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.295330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:18.299264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.299514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.299810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] PATH_INFO: `/v3/projects/bbad5980e6ad45aba5b72ff0b01d5c6b/users/47e6a08ab9584b3abaf3becf7c39e0b9/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.308801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bbad5980e6ad45aba5b72ff0b01d5c6b, user_id=47e6a08ab9584b3abaf3becf7c39e0b9, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:18.315263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-6a6f41b4-db4a-456b-8219-22b8524c08e7 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:18.316016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6a6f41b4-db4a-456b-8219-22b8524c08e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:18.316711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:18.343641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 836/1643] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:18 2022] PUT /identity/v3/projects/bbad5980e6ad45aba5b72ff0b01d5c6b/users/47e6a08ab9584b3abaf3becf7c39e0b9/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:18.344790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:18.453806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 808/1644] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:18:18 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:18.455424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99de726-2766-48ab-9d83-05575c0a20fd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.455756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99de726-2766-48ab-9d83-05575c0a20fd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.456188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99de726-2766-48ab-9d83-05575c0a20fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.502625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cc6b6d-8461-4713-a893-5354c07bb206 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:18.918934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:18.918934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.918934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.918934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] PATH_INFO: `/v3/projects/9fbca917b8174b7c90db70744798b0ba/users/5986abaea4514de0b1b226ed76f08bb8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.919905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9fbca917b8174b7c90db70744798b0ba, user_id=5986abaea4514de0b1b226ed76f08bb8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:18.919905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:18.920403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e99de726-2766-48ab-9d83-05575c0a20fd None None] MFA Rules not processed for user `47e6a08ab9584b3abaf3becf7c39e0b9`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:18.920403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e99de726-2766-48ab-9d83-05575c0a20fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:18.921578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 837/1645] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:18 2022] PUT /identity/v3/projects/9fbca917b8174b7c90db70744798b0ba/users/5986abaea4514de0b1b226ed76f08bb8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:18.921578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:18.923304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 809/1646] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:18 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:18.923304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99de726-2766-48ab-9d83-05575c0a20fd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:18.924166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-812ba11c-7101-4a01-80cb-69e974e02803 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:18.924166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:18.924166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.924166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.925083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] PATH_INFO: `/v3/users/a5b84dec5a8d4a47bb4712d03819affc` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.925083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] RBAC: Authorizing `identity:delete_user(user_id=a5b84dec5a8d4a47bb4712d03819affc)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:18.925083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: WARNING py.warnings [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:18:18.925083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: warnings.warn(msg) Jul 14 12:18:18.925083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: Jul 14 12:18:18.925083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:18.926205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99de726-2766-48ab-9d83-05575c0a20fd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:18.926205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18e1194-29c0-44bb-b0e4-1cf63023c5a2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:18.926205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e1194-29c0-44bb-b0e4-1cf63023c5a2 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:18.926205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e1194-29c0-44bb-b0e4-1cf63023c5a2 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:18.926826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18e1194-29c0-44bb-b0e4-1cf63023c5a2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:18.997693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e1194-29c0-44bb-b0e4-1cf63023c5a2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:19.000268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b18e1194-29c0-44bb-b0e4-1cf63023c5a2 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:19.018870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 810/1647] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:18 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:19.020319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f98fa-c613-46ed-bff2-71b141bd17c3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:19.020748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f98fa-c613-46ed-bff2-71b141bd17c3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:19.021032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460f98fa-c613-46ed-bff2-71b141bd17c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:19.062806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-460f98fa-c613-46ed-bff2-71b141bd17c3 None None] MFA Rules not processed for user `5986abaea4514de0b1b226ed76f08bb8`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:19.064298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-460f98fa-c613-46ed-bff2-71b141bd17c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:19.172063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 811/1648] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:19 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:19.230748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a5b84dec5a8d4a47bb4712d03819affc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:19.331776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a5b84dec5a8d4a47bb4712d03819affc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:19.350802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a5b84dec5a8d4a47bb4712d03819affc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:19.352967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a5b84dec5a8d4a47bb4712d03819affc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:19.356300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a5b84dec5a8d4a47bb4712d03819affc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:19.434972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 838/1649] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:18 2022] DELETE /identity/v3/users/a5b84dec5a8d4a47bb4712d03819affc => generated 0 bytes in 857 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:19.440510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:19.441424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:19.441927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:19.478341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:19.484002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:19.568234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 812/1650] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:19.578639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:19.580088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:19.724084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e84380ee-cebd-430d-bd43-983ba3547dfe admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:19.737149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:19.741214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:19.742191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:19.743129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:19.745680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:19.776085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:19.779669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:19.816593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 839/1651] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 238 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:19.824874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:19.827136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:19.967725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556f9341-e7e3-4438-92f7-b3a97656e69c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:19.972429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:19.974118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:19.974630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:19.975232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:19.981591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:20.066288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:20.075714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:20.079161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:20.101552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 813/1652] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:19 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 277 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:20.259978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:20.273019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa88a426-be6f-48cd-afd6-ac245147b225 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:20.282620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:20.283323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:20.283682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:20.283985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:20.289455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:20.435097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:20.436521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:20.460969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 840/1653] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:20 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:20.567180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:20.902170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:20.907155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:20.927334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a909255-81f6-4a23-9364-1a45d9ac4207 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:20.941997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc8389c-7d3d-4d48-b5f4-13804cbe2b3f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:20.945118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8389c-7d3d-4d48-b5f4-13804cbe2b3f service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:20.945118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8389c-7d3d-4d48-b5f4-13804cbe2b3f service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:20.945565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8389c-7d3d-4d48-b5f4-13804cbe2b3f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:20.948109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5bc8389c-7d3d-4d48-b5f4-13804cbe2b3f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:21.177836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc8389c-7d3d-4d48-b5f4-13804cbe2b3f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:21.180645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc8389c-7d3d-4d48-b5f4-13804cbe2b3f service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:21.219865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 814/1654] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:20 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:21.306107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:21.308659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:21.382829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c461c2bb-0f92-4de8-859c-c54fbd556b04 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:21.405155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:21.406577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:21.409681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:21.410007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] PATH_INFO: `/v3/projects/8b9dfc22855a4c83883bd69d52ffdac1` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:21.413756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b9dfc22855a4c83883bd69d52ffdac1)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:21.421922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:21.432296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:21.534894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:21.539212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:21.539640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:21.590736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] MFA Rules not processed for user `d6da4b0616034aa1bb22c44a8f5ef280`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:21.598073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:21.693103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b9dfc22855a4c83883bd69d52ffdac1', 'request_id': 'req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:21.748785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 815/1655] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:21 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:21.771509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:21.771509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:21.875322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b9dfc22855a4c83883bd69d52ffdac1', 'request_id': 'req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:21.876979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd52bbba-8620-4d51-9d7f-4713e22ca811 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:21.887737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b9dfc22855a4c83883bd69d52ffdac1', 'request_id': 'req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:21.907682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] The token cache is being invalidate because project 8b9dfc22855a4c83883bd69d52ffdac1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:21.908003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:21.908795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:21.911203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 841/1656] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:21 2022] DELETE /identity/v3/projects/8b9dfc22855a4c83883bd69d52ffdac1 => generated 0 bytes in 606 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:21.915187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:21.917041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:21.920277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:21.955766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:21.956804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:21.957271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:21.957699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:21.959823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:22.005559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4881c3a-d8b5-4f19-ba94-87d85d184a9a admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:22.010943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:22.012189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:22.012641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:22.013910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] PATH_INFO: `/v3/users/929c363a7df742748245b912602a7ed2` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:22.015704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=929c363a7df742748245b912602a7ed2)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:22.018227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:22.030665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:22.032411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:22.050290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 816/1657] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:21 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:22.351903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '929c363a7df742748245b912602a7ed2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:22.491229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '929c363a7df742748245b912602a7ed2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:22.504280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '929c363a7df742748245b912602a7ed2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:22.505268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '929c363a7df742748245b912602a7ed2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:22.505937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-405f355c-08c4-4f0b-a7c3-6693ca3611e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '929c363a7df742748245b912602a7ed2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:22.548996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 842/1658] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:21 2022] DELETE /identity/v3/users/929c363a7df742748245b912602a7ed2 => generated 0 bytes in 634 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:22.724976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:22.727233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:22.775654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85990e27-3349-4302-b05c-7515496b5bac service nova] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:22.891511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:22.984178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:22.985067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:22.985574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:22.985853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] PATH_INFO: `/v3/users/414d54ec8e2e466596ce921a8dadde3a` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:22.990462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] RBAC: Authorizing `identity:delete_user(user_id=414d54ec8e2e466596ce921a8dadde3a)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:22.992997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:23.308094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '414d54ec8e2e466596ce921a8dadde3a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:23.399500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '414d54ec8e2e466596ce921a8dadde3a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:23.404638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '414d54ec8e2e466596ce921a8dadde3a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:23.413383 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '414d54ec8e2e466596ce921a8dadde3a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:23.419657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '414d54ec8e2e466596ce921a8dadde3a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:23.459010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 817/1659] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:22 2022] DELETE /identity/v3/users/414d54ec8e2e466596ce921a8dadde3a => generated 0 bytes in 734 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:23.465170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:23.465662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:23.465734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:23.516905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:23.522357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:23.673366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 843/1660] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:23.683093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:23.685687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:23.704467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:23.706245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:23.743513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6114aa95-14ca-4df1-a5f0-c7b31d53a7fd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:23.759677 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:23.760780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:23.761196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:23.761944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] PATH_INFO: `/v3/projects/b745718edc06441e9c0dfbcd98673e23` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:23.764188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b745718edc06441e9c0dfbcd98673e23)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:23.766356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:23.790000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:23.821550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-028df855-f6aa-4a78-bca9-50a79e57f70b admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:23.828708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4473ecc1-0fd8-4383-af3d-13b763331b57 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:23.829909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4473ecc1-0fd8-4383-af3d-13b763331b57 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:23.830571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4473ecc1-0fd8-4383-af3d-13b763331b57 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:23.831268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4473ecc1-0fd8-4383-af3d-13b763331b57 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:23.862305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4473ecc1-0fd8-4383-af3d-13b763331b57 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:23.865340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4473ecc1-0fd8-4383-af3d-13b763331b57 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:23.893936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 818/1661] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:18:23.956971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b745718edc06441e9c0dfbcd98673e23', 'request_id': 'req-2179da41-7a4d-41dd-ae92-caa578304ce9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.062409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b745718edc06441e9c0dfbcd98673e23', 'request_id': 'req-2179da41-7a4d-41dd-ae92-caa578304ce9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.077252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b745718edc06441e9c0dfbcd98673e23', 'request_id': 'req-2179da41-7a4d-41dd-ae92-caa578304ce9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.114573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] The token cache is being invalidate because project b745718edc06441e9c0dfbcd98673e23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:24.114892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.115695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2179da41-7a4d-41dd-ae92-caa578304ce9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.117589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 844/1662] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:23 2022] DELETE /identity/v3/projects/b745718edc06441e9c0dfbcd98673e23 => generated 0 bytes in 414 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:24.122790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2c24d6-9cda-468a-a729-fff69caf7136 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:24.123104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2c24d6-9cda-468a-a729-fff69caf7136 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:24.123394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2c24d6-9cda-468a-a729-fff69caf7136 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:24.156330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3b2c24d6-9cda-468a-a729-fff69caf7136 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:24.158508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3b2c24d6-9cda-468a-a729-fff69caf7136 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:24.315196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 819/1663] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:24.323529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cec377-96e1-415e-927c-3264f808b57a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:24.323841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cec377-96e1-415e-927c-3264f808b57a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:24.324127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5cec377-96e1-415e-927c-3264f808b57a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:24.360524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a5cec377-96e1-415e-927c-3264f808b57a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:24.360524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a5cec377-96e1-415e-927c-3264f808b57a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:24.369891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 845/1664] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:24.374014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2c24d6-9cda-468a-a729-fff69caf7136 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:24.386474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2c24d6-9cda-468a-a729-fff69caf7136 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:24.393537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:24.394276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:24.394564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:24.394869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:24.395633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:24.398125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:24.399886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:24.416255 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 820/1665] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:18:24.421071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cec377-96e1-415e-927c-3264f808b57a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:24.437125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5cec377-96e1-415e-927c-3264f808b57a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:24.442529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:24.443784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:24.444403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:24.444872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:24.445773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:24.448081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:24.535071 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:24.537354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:24.554548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40667720-9f7f-4bb5-9d49-99e9f24686e4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:24.560584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:24.561874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:24.562353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:24.562889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] PATH_INFO: `/v3/users/89f09a31bcaa4553b210a9222c3a794d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:24.564358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89f09a31bcaa4553b210a9222c3a794d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:24.566576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:24.601825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 846/1666] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:24 2022] POST /identity/v3/projects => generated 375 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:24.607071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:24.814893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b68249f8-00a9-4f7e-8165-0c02fb1639b1 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:24.823187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:24.824104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:24.824651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:24.825974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:24.827239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:24.829620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:24.873405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89f09a31bcaa4553b210a9222c3a794d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.921983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:24.935289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 847/1667] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:24 2022] POST /identity/v3/users => generated 288 bytes in 328 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:24.940011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:24.957498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d354e210-5c82-4991-8ed6-992c72545644 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:24.967491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:24.968192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:24.968479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:24.968761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:24.971569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:24.973229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:24.982468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89f09a31bcaa4553b210a9222c3a794d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.985705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89f09a31bcaa4553b210a9222c3a794d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.994082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89f09a31bcaa4553b210a9222c3a794d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.996420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ab2f0c0c-5e91-4abf-b883-259780da6987 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89f09a31bcaa4553b210a9222c3a794d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:24.998604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 848/1668] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:24 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:25.002902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:25.004918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:25.037411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d32ff2-8b05-48c1-b2fa-d990b28a6fee admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:25.037773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 821/1669] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:24 2022] DELETE /identity/v3/users/89f09a31bcaa4553b210a9222c3a794d => generated 0 bytes in 504 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:25.044632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780e078c-a2a1-45ae-86eb-0a307e4bc2da None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:25.046793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780e078c-a2a1-45ae-86eb-0a307e4bc2da None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:25.047095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780e078c-a2a1-45ae-86eb-0a307e4bc2da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:25.093683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-780e078c-a2a1-45ae-86eb-0a307e4bc2da None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:25.099540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-780e078c-a2a1-45ae-86eb-0a307e4bc2da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:25.182895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:25.190356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:25.191660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:25.192048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:25.192519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:25.194613 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:25.199981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:25.221200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 849/1670] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:25 2022] GET /identity/v3/roles => generated 1420 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:25.228063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:25.228063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:25.240171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d900f76d-e962-4af6-b9d1-53026437d32d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:25.246042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:25.247723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:25.248892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:25.249002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 822/1671] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:25.249077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] PATH_INFO: `/v3/projects/78c0c7328a8b4727812b1222e4d5e430` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:25.250196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] RBAC: Authorizing `identity:delete_project(project_id=78c0c7328a8b4727812b1222e4d5e430)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:25.255542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780e078c-a2a1-45ae-86eb-0a307e4bc2da None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:25.259664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:25.281917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780e078c-a2a1-45ae-86eb-0a307e4bc2da None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:25.303478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:25.308600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:25.308600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:25.308600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] PATH_INFO: `/v3/projects/e28c4726590a468490fac2564379e210/users/55b20ca62ddd422ab1722a2bad96e8a9/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:25.347564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] RBAC: Authorizing `identity:create_grant(project_id=e28c4726590a468490fac2564379e210, user_id=55b20ca62ddd422ab1722a2bad96e8a9, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:25.350439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:25.474792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 823/1672] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:25 2022] PUT /identity/v3/projects/e28c4726590a468490fac2564379e210/users/55b20ca62ddd422ab1722a2bad96e8a9/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:25.475890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:25.480221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:25.503875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78c0c7328a8b4727812b1222e4d5e430', 'request_id': 'req-d301be9c-0804-4b7b-8fe3-92489880d23e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:25.550161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9a3026-90cc-4dc9-90d4-1a8cfd4cab5b admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:25.606807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78c0c7328a8b4727812b1222e4d5e430', 'request_id': 'req-d301be9c-0804-4b7b-8fe3-92489880d23e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:25.616860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78c0c7328a8b4727812b1222e4d5e430', 'request_id': 'req-d301be9c-0804-4b7b-8fe3-92489880d23e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:25.632490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:25.633450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:25.633869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:25.634439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:25.645881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] The token cache is being invalidate because project 78c0c7328a8b4727812b1222e4d5e430 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:25.654256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:25.659093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d301be9c-0804-4b7b-8fe3-92489880d23e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:25.665476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 850/1673] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:25 2022] DELETE /identity/v3/projects/78c0c7328a8b4727812b1222e4d5e430 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:25.666972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:25.667610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:25.667982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:25.754823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:25.757189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:25.760331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] MFA Rules not processed for user `55b20ca62ddd422ab1722a2bad96e8a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:25.761760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:25.807216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 824/1674] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:25.808868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:25.823177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a954f61-8bc9-4712-8c02-a2048dce7d84 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:25.832916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:25.834022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:25.834472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:25.835728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] PATH_INFO: `/v3/users/5986abaea4514de0b1b226ed76f08bb8` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:25.837913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=5986abaea4514de0b1b226ed76f08bb8)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:25.840560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:25.940129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 851/1675] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:25 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:25.962492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:25.964245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:26.076422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5dc713-1a15-446d-b9e4-cf06a685a1fb None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:26.085909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:26.086793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:26.087884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:26.087884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:26.101730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:26.103521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:26.127769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 852/1676] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:25 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:18:26.195931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5986abaea4514de0b1b226ed76f08bb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:26.285729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5986abaea4514de0b1b226ed76f08bb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:26.286527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5986abaea4514de0b1b226ed76f08bb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:26.306996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5986abaea4514de0b1b226ed76f08bb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:26.309246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5986abaea4514de0b1b226ed76f08bb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:26.342683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 825/1677] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:25 2022] DELETE /identity/v3/users/5986abaea4514de0b1b226ed76f08bb8 => generated 0 bytes in 532 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:27.647147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:27.648684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:27.684274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e97fdf-fcc5-45a8-9d0a-547016c85227 service glance] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:27.787981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:27.789567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:27.790025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:27.790443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] PATH_INFO: `/v3/projects/e24bf8b5dd8247039f6a5ac09db96698` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:27.791853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e24bf8b5dd8247039f6a5ac09db96698)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:27.795681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:27.933773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:27.934962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:27.949928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:27.974295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ed1966-9a82-43e9-9b8f-477b5a0d56eb admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:28.005814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e24bf8b5dd8247039f6a5ac09db96698', 'request_id': 'req-38d62a5e-2192-4cfe-be3c-44a842a8e239'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.102401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e24bf8b5dd8247039f6a5ac09db96698', 'request_id': 'req-38d62a5e-2192-4cfe-be3c-44a842a8e239'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.113163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e24bf8b5dd8247039f6a5ac09db96698', 'request_id': 'req-38d62a5e-2192-4cfe-be3c-44a842a8e239'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.150974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] The token cache is being invalidate because project e24bf8b5dd8247039f6a5ac09db96698 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:28.151273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.155933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-38d62a5e-2192-4cfe-be3c-44a842a8e239 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.165839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 853/1678] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:27 2022] DELETE /identity/v3/projects/e24bf8b5dd8247039f6a5ac09db96698 => generated 0 bytes in 516 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:28.181886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad523f44-a3c0-45c6-b33d-c82aab589691 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:28.182521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad523f44-a3c0-45c6-b33d-c82aab589691 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:28.185743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad523f44-a3c0-45c6-b33d-c82aab589691 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:28.187563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:28.188452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:28.190133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:28.190133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] PATH_INFO: `/v3/projects/9fbca917b8174b7c90db70744798b0ba` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:28.191804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9fbca917b8174b7c90db70744798b0ba)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:28.192858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: WARNING py.warnings [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 14 12:18:28.192858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: warnings.warn(msg) Jul 14 12:18:28.192858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: Jul 14 12:18:28.194063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:28.260103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ad523f44-a3c0-45c6-b33d-c82aab589691 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:28.271095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ad523f44-a3c0-45c6-b33d-c82aab589691 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:28.298488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 854/1679] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:28.305502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1fb3e4-0660-451f-a5d6-7465ae8089b2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:28.305902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1fb3e4-0660-451f-a5d6-7465ae8089b2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:28.306318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f1fb3e4-0660-451f-a5d6-7465ae8089b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:28.365568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1f1fb3e4-0660-451f-a5d6-7465ae8089b2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:28.369943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1f1fb3e4-0660-451f-a5d6-7465ae8089b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:28.394896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 855/1680] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:28.415860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1fb3e4-0660-451f-a5d6-7465ae8089b2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:28.435176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:28.463547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f1fb3e4-0660-451f-a5d6-7465ae8089b2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:28.505839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9fbca917b8174b7c90db70744798b0ba', 'request_id': 'req-e01e1478-a65d-4747-b474-d36dbe8fee61'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.544353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9fbca917b8174b7c90db70744798b0ba', 'request_id': 'req-e01e1478-a65d-4747-b474-d36dbe8fee61'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.635014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff33ef13-26cb-4c54-8c7c-52368b8cc488 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:28.635754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33ef13-26cb-4c54-8c7c-52368b8cc488 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:28.636044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33ef13-26cb-4c54-8c7c-52368b8cc488 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:28.636325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff33ef13-26cb-4c54-8c7c-52368b8cc488 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:28.637517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff33ef13-26cb-4c54-8c7c-52368b8cc488 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:28.639885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff33ef13-26cb-4c54-8c7c-52368b8cc488 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:28.643412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff33ef13-26cb-4c54-8c7c-52368b8cc488 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:28.665118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 856/1681] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:28.671937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c5daf-d634-4fec-a008-20579ed6b559 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:28.672401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c5daf-d634-4fec-a008-20579ed6b559 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:28.672864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-006c5daf-d634-4fec-a008-20579ed6b559 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:28.696630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9fbca917b8174b7c90db70744798b0ba', 'request_id': 'req-e01e1478-a65d-4747-b474-d36dbe8fee61'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.755641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] The token cache is being invalidate because project 9fbca917b8174b7c90db70744798b0ba was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:28.756945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.762288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:28.766851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 826/1682] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:27 2022] DELETE /identity/v3/projects/9fbca917b8174b7c90db70744798b0ba => generated 0 bytes in 834 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:28.768207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:28.773496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:28.800283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-006c5daf-d634-4fec-a008-20579ed6b559 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:28.801832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-006c5daf-d634-4fec-a008-20579ed6b559 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:28.813287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01e1478-a65d-4747-b474-d36dbe8fee61 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:28.821019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:28.821796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:28.822302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:28.822597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:28.823551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:28.826456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:28.960918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 857/1683] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:18:28 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 293 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:28.962258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f3fc3-f2dd-47f2-a30a-6b23553bf6f5 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:28.962553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f3fc3-f2dd-47f2-a30a-6b23553bf6f5 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:28.962862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128f3fc3-f2dd-47f2-a30a-6b23553bf6f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.003841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-128f3fc3-f2dd-47f2-a30a-6b23553bf6f5 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:29.005218 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-128f3fc3-f2dd-47f2-a30a-6b23553bf6f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:29.014987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 858/1684] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:29.024722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e636b19-b618-4f27-b728-63208732fcf3 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.025013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e636b19-b618-4f27-b728-63208732fcf3 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.025350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e636b19-b618-4f27-b728-63208732fcf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.051651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8e636b19-b618-4f27-b728-63208732fcf3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:29.056735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8e636b19-b618-4f27-b728-63208732fcf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:29.067062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 859/1685] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:29.070828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e636b19-b618-4f27-b728-63208732fcf3 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.083197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e636b19-b618-4f27-b728-63208732fcf3 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.088670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:29.089703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.090035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.090389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.091736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:29.092048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:29.093748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:29.107194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 860/1686] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:18:29.113383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.121905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 827/1687] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:28 2022] POST /identity/v3/projects => generated 367 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:29.123615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.125439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3352047-394f-46d6-860e-e7fb40189dc6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.240052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:29.240744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.241032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.241311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.242815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:29.244035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ddd131-837d-4ddf-9e0f-64287319e050 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.244798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:29.250281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daf7bcdd-25fa-417e-bf54-9659dd18117b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:29.251381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf7bcdd-25fa-417e-bf54-9659dd18117b admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.251999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf7bcdd-25fa-417e-bf54-9659dd18117b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.252422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daf7bcdd-25fa-417e-bf54-9659dd18117b admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.253270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf7bcdd-25fa-417e-bf54-9659dd18117b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:29.255751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daf7bcdd-25fa-417e-bf54-9659dd18117b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:29.376445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 828/1688] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:29 2022] POST /identity/v3/users => generated 284 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:29.382117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a630e7-58f9-4067-9315-350315b2ebf6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.382575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a630e7-58f9-4067-9315-350315b2ebf6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.383225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a630e7-58f9-4067-9315-350315b2ebf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.402987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 861/1689] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:29 2022] POST /identity/v3/projects => generated 385 bytes in 288 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:29.402987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.423923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d5a630e7-58f9-4067-9315-350315b2ebf6 None None] MFA Rules not processed for user `d6da4b0616034aa1bb22c44a8f5ef280`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:29.425946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d5a630e7-58f9-4067-9315-350315b2ebf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:29.578174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34d850dc-115e-4e51-9f14-e8f5db207a99 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.603470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:29.603913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.604369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.604657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.607358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:29.607681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 829/1690] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:29 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:29.609265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a630e7-58f9-4067-9315-350315b2ebf6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.624924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:29.640579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a630e7-58f9-4067-9315-350315b2ebf6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.660746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:29.661464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.661747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.662023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.662723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:29.668347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:29.671153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 862/1691] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:29 2022] GET /identity/v3/roles => generated 1420 bytes in 270 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:29.672154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.681475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:29.803839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 830/1692] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:29 2022] POST /identity/v3/users => generated 293 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:29.803839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.856214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c8bd26-b532-4901-b977-930b8ffad679 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.877868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:29.878683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.879156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.879532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.881125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:29.890879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:29.920657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 831/1693] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:29 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:29.921979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:29.934170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea02ade8-6512-4af1-a0bb-faf6fbb0ecee admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.940475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:29.941765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:29.942253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:29.942726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:29.943748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:29.945627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:29.989582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 832/1694] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:29 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:29.989950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0becc546-5b7b-41da-a16c-e150c2d61b2b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:29.991915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:30.006168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:30.008066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.008066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.008066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.024200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ecc9956-348b-4ff3-9e31-11873f12d5b9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:30.034262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:30.035338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.035562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.035842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] PATH_INFO: `/v3/projects/52b69bf6562e4a25800f13288975e44e/users/de537197766d475a82190f876953e934/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.086719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52b69bf6562e4a25800f13288975e44e, user_id=de537197766d475a82190f876953e934, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:30.089532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:30.129868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:30.131467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:30.146583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 863/1695] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:18:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 475 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:18:30.147695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:30.161068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc38dd3-1bd3-41fb-89e4-e788fa6fa435 service swift] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:30.170565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c6c266b-c522-4827-a7a7-5f52d31fb7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:30.170565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6c266b-c522-4827-a7a7-5f52d31fb7d8 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.170565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6c266b-c522-4827-a7a7-5f52d31fb7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.170565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c6c266b-c522-4827-a7a7-5f52d31fb7d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.170950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6c266b-c522-4827-a7a7-5f52d31fb7d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:30.172046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c6c266b-c522-4827-a7a7-5f52d31fb7d8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:30.199534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 833/1696] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:29 2022] PUT /identity/v3/projects/52b69bf6562e4a25800f13288975e44e/users/de537197766d475a82190f876953e934/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:30.211227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 864/1697] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:30 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:30.211326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:30.212747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.213111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.213457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.280105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] MFA Rules not processed for user `de537197766d475a82190f876953e934`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:30.281292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:30.297754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:30.348775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49403528-8906-4dd6-834d-02526e93fe69 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:30.354978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3a8717-7166-4ea5-9d10-ba7650eacc9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:30.357321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a8717-7166-4ea5-9d10-ba7650eacc9a admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.357952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a8717-7166-4ea5-9d10-ba7650eacc9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.358382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3a8717-7166-4ea5-9d10-ba7650eacc9a admin admin] PATH_INFO: `/v3/projects/d275cd8750a84a94ab24537e76c4b075/users/b0ac2ce0059b43f190cdb3b38652fcda/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.389229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3a8717-7166-4ea5-9d10-ba7650eacc9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=d275cd8750a84a94ab24537e76c4b075, user_id=b0ac2ce0059b43f190cdb3b38652fcda, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:30.392324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3a8717-7166-4ea5-9d10-ba7650eacc9a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:30.449480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 865/1698] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:30 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:30.462066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:30.469842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:30.493844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 834/1699] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:30 2022] PUT /identity/v3/projects/d275cd8750a84a94ab24537e76c4b075/users/b0ac2ce0059b43f190cdb3b38652fcda/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 283 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:30.507432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b74bb5a-1d16-4e10-98a8-8a55130a9c1d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.507981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b74bb5a-1d16-4e10-98a8-8a55130a9c1d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.508420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b74bb5a-1d16-4e10-98a8-8a55130a9c1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.571367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-8b74bb5a-1d16-4e10-98a8-8a55130a9c1d None None] MFA Rules not processed for user `b0ac2ce0059b43f190cdb3b38652fcda`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:30.573157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8b74bb5a-1d16-4e10-98a8-8a55130a9c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:30.597131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147f3c89-d350-4834-814a-cff00e446bcd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:30.617857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:30.730675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf19176-fcf8-4d27-94b2-73c749734602 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:30.859270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf19176-fcf8-4d27-94b2-73c749734602 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.859270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf19176-fcf8-4d27-94b2-73c749734602 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.859270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf19176-fcf8-4d27-94b2-73c749734602 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.860218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 835/1700] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:30 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:30.860218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21d342-3a60-4e8a-9481-dbb2551c33e5 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.860218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21d342-3a60-4e8a-9481-dbb2551c33e5 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.860218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e21d342-3a60-4e8a-9481-dbb2551c33e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.860218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9e21d342-3a60-4e8a-9481-dbb2551c33e5 None None] MFA Rules not processed for user `0d302e57946c42d799e9bf120873eba6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:30.860218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9e21d342-3a60-4e8a-9481-dbb2551c33e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:30.861116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf19176-fcf8-4d27-94b2-73c749734602 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:30.861116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf19176-fcf8-4d27-94b2-73c749734602 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:30.861116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 866/1701] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:30 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:30.861116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d73322-0840-4861-82ab-c0184cfda8d7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.861116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d73322-0840-4861-82ab-c0184cfda8d7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.861116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4d73322-0840-4861-82ab-c0184cfda8d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:30.888014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d4d73322-0840-4861-82ab-c0184cfda8d7 None None] MFA Rules not processed for user `a0a6364a4a884b09b5eb80897c19e296`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:30.889357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d4d73322-0840-4861-82ab-c0184cfda8d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:30.890896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 836/1702] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:30 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:30.892363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a554a84b-9221-4073-a315-1b61b67bc0e7 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:30.894636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a554a84b-9221-4073-a315-1b61b67bc0e7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:30.895027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a554a84b-9221-4073-a315-1b61b67bc0e7 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.026707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 837/1703] 10.222.0.27 () {54 vars in 912 bytes} [Thu Jul 14 12:18:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:31.026707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a554a84b-9221-4073-a315-1b61b67bc0e7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:31.026707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a554a84b-9221-4073-a315-1b61b67bc0e7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:31.027535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 867/1704] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:30 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:31.027535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160bac6e-4df6-4832-a7ac-a5363cd37342 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.027535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160bac6e-4df6-4832-a7ac-a5363cd37342 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.027535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-160bac6e-4df6-4832-a7ac-a5363cd37342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.027787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:31.027787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.027787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.027787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.028054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:31.028054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:31.065331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-160bac6e-4df6-4832-a7ac-a5363cd37342 None None] MFA Rules not processed for user `0d302e57946c42d799e9bf120873eba6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:31.069089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-160bac6e-4df6-4832-a7ac-a5363cd37342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:31.082742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 868/1705] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:31.084182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92d8d84-9ad1-4cc3-80ad-ccf2f2f49d86 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.084536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92d8d84-9ad1-4cc3-80ad-ccf2f2f49d86 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.084885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92d8d84-9ad1-4cc3-80ad-ccf2f2f49d86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.132650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a92d8d84-9ad1-4cc3-80ad-ccf2f2f49d86 None None] MFA Rules not processed for user `a4fff90ae5434b58932f468ca9c827f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:31.134049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a92d8d84-9ad1-4cc3-80ad-ccf2f2f49d86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:31.190833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 869/1706] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:18:31 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:31.196143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.196621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.196900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.216820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] MFA Rules not processed for user `a0a6364a4a884b09b5eb80897c19e296`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:31.218192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:31.225603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 870/1707] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:31 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:31.226589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:31.231365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:31.327185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 838/1708] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:30 2022] POST /identity/v3/projects => generated 385 bytes in 429 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:31.333298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:31.333298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:31.349395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc1ca29-3326-4233-b810-dda93216c414 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:31.433984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:31.435160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.435574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.436144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.442620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74d2921-4a62-4521-b3aa-f6b1996856df admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:31.453181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:31.454454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.454454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.455751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.467189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:31.468807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:31.480013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 839/1709] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:31 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:31.486901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:31.542074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:31.545739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:31.567547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 871/1710] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:31 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:31.575471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da460fd8-fb3d-417b-8ecc-a85186ef37df service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:31.580866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:31.581694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.582092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.582466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.583507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:31.585608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:31.779945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 840/1711] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:31 2022] POST /identity/v3/users => generated 299 bytes in 294 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:31.780099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:31.799821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7243637a-06df-4d82-be47-2f4e1df77760 service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:31.822038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:31.823057 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.823498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.823925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.825602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:31.828932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:31.847848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 872/1712] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:31 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:31.848880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:31.877211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bdd0fc-5bb8-4e90-8c22-91d9006cc39a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:31.902057 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:31.904400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:31.905772 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:31.905772 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] PATH_INFO: `/v3/projects/38fd4e9f0be14117a09e24bc292f11d7/users/dc04c7c913cc464b9c33ec6ab34323ca/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:31.939916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=38fd4e9f0be14117a09e24bc292f11d7, user_id=dc04c7c913cc464b9c33ec6ab34323ca, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:31.942849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:32.065587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 841/1713] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:31 2022] PUT /identity/v3/projects/38fd4e9f0be14117a09e24bc292f11d7/users/dc04c7c913cc464b9c33ec6ab34323ca/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:32.066908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:32.084733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b179f6-09bd-4693-a6a5-0737a290a7eb admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:32.253383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8660312c-1bba-4fc7-b768-297f0271484b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:32.254166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660312c-1bba-4fc7-b768-297f0271484b admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.254577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660312c-1bba-4fc7-b768-297f0271484b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.254952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8660312c-1bba-4fc7-b768-297f0271484b admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.255947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8660312c-1bba-4fc7-b768-297f0271484b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:32.258600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8660312c-1bba-4fc7-b768-297f0271484b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:32.274121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 873/1714] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:32 2022] GET /identity/v3/roles => generated 1420 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:32.279242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:32.297278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43d047b-ff65-4654-b9f8-a04afae5e0f6 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:32.307340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86672f5-be94-4c81-8aaa-dfdb116adc4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:32.307340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86672f5-be94-4c81-8aaa-dfdb116adc4c admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.307340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86672f5-be94-4c81-8aaa-dfdb116adc4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.307340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86672f5-be94-4c81-8aaa-dfdb116adc4c admin admin] PATH_INFO: `/v3/projects/38fd4e9f0be14117a09e24bc292f11d7/users/dc04c7c913cc464b9c33ec6ab34323ca/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.307946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86672f5-be94-4c81-8aaa-dfdb116adc4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=38fd4e9f0be14117a09e24bc292f11d7, user_id=dc04c7c913cc464b9c33ec6ab34323ca, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:32.311631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86672f5-be94-4c81-8aaa-dfdb116adc4c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:32.326849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 842/1715] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:32 2022] PUT /identity/v3/projects/38fd4e9f0be14117a09e24bc292f11d7/users/dc04c7c913cc464b9c33ec6ab34323ca/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:32.335207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd458b-a694-446b-b813-cad8f27c0020 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.335412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd458b-a694-446b-b813-cad8f27c0020 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.335713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cfd458b-a694-446b-b813-cad8f27c0020 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.380503 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4cfd458b-a694-446b-b813-cad8f27c0020 None None] MFA Rules not processed for user `dc04c7c913cc464b9c33ec6ab34323ca`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:32.382190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4cfd458b-a694-446b-b813-cad8f27c0020 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:32.478121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a47ee6-430d-4f51-a0fe-cf475bbe8ea0 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.478655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a47ee6-430d-4f51-a0fe-cf475bbe8ea0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.478920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a47ee6-430d-4f51-a0fe-cf475bbe8ea0 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.480424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 843/1716] 10.222.0.27 () {54 vars in 912 bytes} [Thu Jul 14 12:18:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:18:32.486160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b1a36-6401-45f8-917b-3149d8e24a57 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.486740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b1a36-6401-45f8-917b-3149d8e24a57 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.487350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6b1a36-6401-45f8-917b-3149d8e24a57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.528126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-be6b1a36-6401-45f8-917b-3149d8e24a57 None None] MFA Rules not processed for user `a4fff90ae5434b58932f468ca9c827f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:32.529989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-be6b1a36-6401-45f8-917b-3149d8e24a57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:32.543477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 874/1717] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:32 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:32.543477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55988329-21df-4220-b83f-416051450789 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.543477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55988329-21df-4220-b83f-416051450789 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.566091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55988329-21df-4220-b83f-416051450789 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.566091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 875/1718] 10.222.0.27 () {54 vars in 960 bytes} [Thu Jul 14 12:18:32 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:32.566091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55988329-21df-4220-b83f-416051450789 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:32.566091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-55988329-21df-4220-b83f-416051450789 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:32.672775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 844/1719] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:18:32 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:32.674298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.674590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.674892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.715405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55988329-21df-4220-b83f-416051450789 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:32.720212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:32.721476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.721764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.722148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.724351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:32.726015 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:32.848225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 845/1720] 10.222.0.27 () {58 vars in 1057 bytes} [Thu Jul 14 12:18:32 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:32.849691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:32.851285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:32.863944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:32.865772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:32.887259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 876/1721] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:32 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:32.888348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:32.894952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:32.931735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f0f4a96-5afe-4b52-ae78-3db89e2601f5 service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:32.933623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e29363c-ca92-487a-9b69-d6e218e7c983 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:32.938250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ceb237-ce4e-4e3e-887f-45fe9c5dfb49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:32.939658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ceb237-ce4e-4e3e-887f-45fe9c5dfb49 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.939961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ceb237-ce4e-4e3e-887f-45fe9c5dfb49 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.940253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ceb237-ce4e-4e3e-887f-45fe9c5dfb49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.943460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:32.944643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:32.944939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:32.945219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:32.956379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:32.959089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:32.966548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ceb237-ce4e-4e3e-887f-45fe9c5dfb49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:32.968346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2ceb237-ce4e-4e3e-887f-45fe9c5dfb49 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:32.979425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 877/1722] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:32 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:32.982105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:32.992774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 846/1723] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:32 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:33.019070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d8021d-1903-4244-a0e8-411e3d121229 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:33.045390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:33.049962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:33.049962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:33.049962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:33.052065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:33.086164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:33.090552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:33.139353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 878/1724] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:32 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:33.366597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:34.190311 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d4c3fe-1a07-4165-88d6-216139a7afb3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:34.190977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d4c3fe-1a07-4165-88d6-216139a7afb3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:34.192312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d4c3fe-1a07-4165-88d6-216139a7afb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:34.217688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-57d4c3fe-1a07-4165-88d6-216139a7afb3 None None] MFA Rules not processed for user `0d302e57946c42d799e9bf120873eba6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:34.219220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-57d4c3fe-1a07-4165-88d6-216139a7afb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:34.268141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:34.391314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 847/1725] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:34 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:35.652970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:35.654334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:35.843574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2511d68-61eb-4788-b43a-1622931ccaa7 service placement] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:35.862484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:35.863638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:35.863932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:35.864207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] PATH_INFO: `/v3/users/09f5c519c42f484b990b89ad5d3c93d5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:35.867126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=09f5c519c42f484b990b89ad5d3c93d5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:35.869927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:36.235807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09f5c519c42f484b990b89ad5d3c93d5'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.317788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09f5c519c42f484b990b89ad5d3c93d5'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.328725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d4c3fe-1a07-4165-88d6-216139a7afb3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:36.337611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09f5c519c42f484b990b89ad5d3c93d5'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.339094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09f5c519c42f484b990b89ad5d3c93d5'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.341155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e7fa5115-d040-476b-af5b-8c51422dc7ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09f5c519c42f484b990b89ad5d3c93d5'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.346047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d4c3fe-1a07-4165-88d6-216139a7afb3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:36.363772 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:36.365450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:36.368164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:36.369129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] PATH_INFO: `/v3/users/b0ac2ce0059b43f190cdb3b38652fcda` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:36.375720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b0ac2ce0059b43f190cdb3b38652fcda)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:36.380099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:36.409108 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 879/1726] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:35 2022] DELETE /identity/v3/users/09f5c519c42f484b990b89ad5d3c93d5 => generated 0 bytes in 757 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:36.413769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601ea080-f542-4911-b178-9831884a72e3 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:36.414377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601ea080-f542-4911-b178-9831884a72e3 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:36.414833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-601ea080-f542-4911-b178-9831884a72e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:36.446205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-601ea080-f542-4911-b178-9831884a72e3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:36.447749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-601ea080-f542-4911-b178-9831884a72e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:36.534905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 880/1727] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:36.545976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601ea080-f542-4911-b178-9831884a72e3 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:36.568386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-601ea080-f542-4911-b178-9831884a72e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:36.659694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-601ea080-f542-4911-b178-9831884a72e3 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:36.665534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dfb2ddd-2fc2-4163-a83a-d0701d1bc64f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:36.666809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfb2ddd-2fc2-4163-a83a-d0701d1bc64f service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:36.668320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfb2ddd-2fc2-4163-a83a-d0701d1bc64f service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:36.670733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dfb2ddd-2fc2-4163-a83a-d0701d1bc64f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:36.695768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfb2ddd-2fc2-4163-a83a-d0701d1bc64f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:36.700957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dfb2ddd-2fc2-4163-a83a-d0701d1bc64f service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:36.727545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 881/1728] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:18:36.734038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b0ac2ce0059b43f190cdb3b38652fcda'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.863958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b0ac2ce0059b43f190cdb3b38652fcda'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.865125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b0ac2ce0059b43f190cdb3b38652fcda'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.883755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b0ac2ce0059b43f190cdb3b38652fcda'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.884517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b0ac2ce0059b43f190cdb3b38652fcda'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:36.913353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 848/1729] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:36 2022] DELETE /identity/v3/users/b0ac2ce0059b43f190cdb3b38652fcda => generated 0 bytes in 585 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:36.918844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:36.919451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:36.919870 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:36.952399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:36.953860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:37.090880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 882/1730] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:37.107335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:37.123889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:37.269359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a14c5b64-7a7b-465a-b0b9-cde7965fc112 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:37.273845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1521fe2-4f0f-49f5-b516-23c612044347 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:37.274982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1521fe2-4f0f-49f5-b516-23c612044347 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:37.275463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1521fe2-4f0f-49f5-b516-23c612044347 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:37.275866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1521fe2-4f0f-49f5-b516-23c612044347 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:37.309039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1521fe2-4f0f-49f5-b516-23c612044347 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:37.314013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1521fe2-4f0f-49f5-b516-23c612044347 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:37.328614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 849/1731] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 222 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:37.657177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:37.676313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:37.678845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:37.725415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24c3e2e-fdff-435e-85f4-910330ef0d3a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:37.750305 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:37.750305 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:37.750305 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:37.750305 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] PATH_INFO: `/v3/projects/e8c82e1c7ee74798b03deba105d1cbf0` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:37.751020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] RBAC: Authorizing `identity:delete_project(project_id=e8c82e1c7ee74798b03deba105d1cbf0)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:37.751711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:37.935950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e8c82e1c7ee74798b03deba105d1cbf0', 'request_id': 'req-0bf9fc54-91b7-40d9-b952-840da01f737e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.079705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e8c82e1c7ee74798b03deba105d1cbf0', 'request_id': 'req-0bf9fc54-91b7-40d9-b952-840da01f737e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.088697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e8c82e1c7ee74798b03deba105d1cbf0', 'request_id': 'req-0bf9fc54-91b7-40d9-b952-840da01f737e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.104539 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] The token cache is being invalidate because project e8c82e1c7ee74798b03deba105d1cbf0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:38.105056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.105863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.112439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 883/1732] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:37 2022] DELETE /identity/v3/projects/e8c82e1c7ee74798b03deba105d1cbf0 => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:38.117608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14005803-5d4d-4545-a700-cec430c306b7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:38.118076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14005803-5d4d-4545-a700-cec430c306b7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:38.118447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14005803-5d4d-4545-a700-cec430c306b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:38.138868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-14005803-5d4d-4545-a700-cec430c306b7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:38.140159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-14005803-5d4d-4545-a700-cec430c306b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:38.194714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:38.198060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:38.242854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 850/1733] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:38.250337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9200570c-1690-47c4-9bbe-0b0c08b59dbd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:38.250944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9200570c-1690-47c4-9bbe-0b0c08b59dbd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:38.251815 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9200570c-1690-47c4-9bbe-0b0c08b59dbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:38.313568 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9200570c-1690-47c4-9bbe-0b0c08b59dbd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:38.315103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9200570c-1690-47c4-9bbe-0b0c08b59dbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:38.329276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 851/1734] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:38.333907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9200570c-1690-47c4-9bbe-0b0c08b59dbd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:38.339245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf9fc54-91b7-40d9-b952-840da01f737e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:38.347905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:38.348327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:38.348671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:38.348841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9200570c-1690-47c4-9bbe-0b0c08b59dbd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:38.348940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] PATH_INFO: `/v3/projects/d275cd8750a84a94ab24537e76c4b075` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:38.350975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d275cd8750a84a94ab24537e76c4b075)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:38.355021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:38.355900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:38.357697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:38.357697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:38.369794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:38.371437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:38.371936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:38.376773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:38.425292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 852/1735] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:38.429781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:38.431847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:38.450225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4fd3fc-f6dd-4595-9f40-ea8e1bbbb558 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:38.467077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:38.468307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:38.468307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:38.468511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:38.469349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:38.473606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:38.571005 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd275cd8750a84a94ab24537e76c4b075', 'request_id': 'req-235dca5f-6ff0-494a-9814-cee423e19f8b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.585135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 853/1736] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:38 2022] POST /identity/v3/projects => generated 403 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:38.589219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:38.656389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd275cd8750a84a94ab24537e76c4b075', 'request_id': 'req-235dca5f-6ff0-494a-9814-cee423e19f8b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.664592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd275cd8750a84a94ab24537e76c4b075', 'request_id': 'req-235dca5f-6ff0-494a-9814-cee423e19f8b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.694427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] The token cache is being invalidate because project d275cd8750a84a94ab24537e76c4b075 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:38.696268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.697975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:38.705694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 884/1737] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:38 2022] DELETE /identity/v3/projects/d275cd8750a84a94ab24537e76c4b075 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:38.707109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:38.708897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:38.721380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d3ae232-3ea3-4ad0-abf3-4c58fa9432fd admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:38.724095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:38.725220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:38.725600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:38.726088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:38.739428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:38.765390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:38.958726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235dca5f-6ff0-494a-9814-cee423e19f8b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:38.966544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:38.967917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:38.968345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:38.968804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] PATH_INFO: `/v3/users/dc04c7c913cc464b9c33ec6ab34323ca` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:38.975795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dc04c7c913cc464b9c33ec6ab34323ca)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:38.979574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:39.002859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 854/1738] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:38 2022] POST /identity/v3/users => generated 302 bytes in 407 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:39.005308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:39.007082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:39.031069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9739196f-5b32-4c80-b03b-48760b37ece3 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:39.050193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:39.051756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:39.052257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:39.052758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:39.054097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:39.066279 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:39.089900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 855/1739] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:39 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:18:39.097297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:39.137268 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e02af46-f483-40ef-b8da-603bcb28dbe8 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:39.149847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:39.150541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:39.150856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:39.151135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:39.155186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:39.159941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:39.196403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 856/1740] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:39 2022] GET /identity/v3/roles => generated 1420 bytes in 101 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:39.202011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:39.224736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c1060c7-ded1-4434-b856-a346199297fa admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:39.231083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28be8fc1-09ea-47c2-9eaf-f6fda439c32d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:39.233319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be8fc1-09ea-47c2-9eaf-f6fda439c32d admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:39.233983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be8fc1-09ea-47c2-9eaf-f6fda439c32d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:39.234592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28be8fc1-09ea-47c2-9eaf-f6fda439c32d admin admin] PATH_INFO: `/v3/projects/7cdf723129954ef4b166727c84dd9d57/users/a6aa2404a2a4422eb6b5bf4496656c7a/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:39.267127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28be8fc1-09ea-47c2-9eaf-f6fda439c32d admin admin] RBAC: Authorizing `identity:create_grant(project_id=7cdf723129954ef4b166727c84dd9d57, user_id=a6aa2404a2a4422eb6b5bf4496656c7a, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:39.270099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28be8fc1-09ea-47c2-9eaf-f6fda439c32d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:39.332709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dc04c7c913cc464b9c33ec6ab34323ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:39.405874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dc04c7c913cc464b9c33ec6ab34323ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:39.413346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 857/1741] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:39 2022] PUT /identity/v3/projects/7cdf723129954ef4b166727c84dd9d57/users/a6aa2404a2a4422eb6b5bf4496656c7a/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:39.418662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:39.419070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:39.419428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:39.420713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dc04c7c913cc464b9c33ec6ab34323ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:39.423030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dc04c7c913cc464b9c33ec6ab34323ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:39.423775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9896175b-21be-4449-a646-1df684a84003 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dc04c7c913cc464b9c33ec6ab34323ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:39.460323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 885/1742] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:38 2022] DELETE /identity/v3/users/dc04c7c913cc464b9c33ec6ab34323ca => generated 0 bytes in 754 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:39.471366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:39.476136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:39.654670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8506d419-75ca-4312-8b25-5d7d4abb81ef None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:39.659820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8506d419-75ca-4312-8b25-5d7d4abb81ef None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:39.659820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8506d419-75ca-4312-8b25-5d7d4abb81ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:39.660281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 858/1743] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:39.678018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:39.679695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:39.720873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:39.730437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8506d419-75ca-4312-8b25-5d7d4abb81ef None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:39.736006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8506d419-75ca-4312-8b25-5d7d4abb81ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:39.835407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-153de53b-c2c1-4aa4-ad69-16eb09b5aa97 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:39.850263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:39.853072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:39.853382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:39.856359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:39.892705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:39.893113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:39.931831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 859/1744] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:18:39.936403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 886/1745] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:18:39 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:39.952027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:39.954295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:40.052178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d206c57-8338-4667-9c06-81ab6cfa835b service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:40.057648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:40.058972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:40.059768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:40.060355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:40.071109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:40.073313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:40.090194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 860/1746] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:18:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:18:40.098678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:41.911567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:41.912582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:41.913039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:41.966264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] MFA Rules not processed for user `a4fff90ae5434b58932f468ca9c827f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:41.966264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:41.989820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 887/1747] 10.222.0.86 () {58 vars in 1015 bytes} [Thu Jul 14 12:18:41 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:42.011609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:42.014898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:42.040235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:42.051925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cc77cb-fdad-4c98-a0b8-9be4bc0e2fb8 service swift] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:42.067599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:42.068655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:42.069013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:42.069392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] PATH_INFO: `/v3/projects/38fd4e9f0be14117a09e24bc292f11d7` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:42.073993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:42.076333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:42.079332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=38fd4e9f0be14117a09e24bc292f11d7)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:42.087326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:42.108213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156cf45e-2651-4167-90a4-5fb7cb12039f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:42.118819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:42.119495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:42.119781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:42.120067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:42.169467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:42.172840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:42.230917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 888/1748] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:42 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:42.232390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:42.253730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0aa823d-aebf-48a7-831c-a3893061d2b5 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:42.258623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0347d15a-9feb-46ae-96b8-d3c810fd2fcb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:42.263203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0347d15a-9feb-46ae-96b8-d3c810fd2fcb service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:42.263203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0347d15a-9feb-46ae-96b8-d3c810fd2fcb service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:42.263203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0347d15a-9feb-46ae-96b8-d3c810fd2fcb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:42.284769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0347d15a-9feb-46ae-96b8-d3c810fd2fcb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:42.284769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0347d15a-9feb-46ae-96b8-d3c810fd2fcb service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:42.321214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 889/1749] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:42 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:42.365308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '38fd4e9f0be14117a09e24bc292f11d7', 'request_id': 'req-3805a745-ac4b-44e9-bafa-844e5a15cb1c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:42.368828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '38fd4e9f0be14117a09e24bc292f11d7', 'request_id': 'req-3805a745-ac4b-44e9-bafa-844e5a15cb1c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:42.491153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '38fd4e9f0be14117a09e24bc292f11d7', 'request_id': 'req-3805a745-ac4b-44e9-bafa-844e5a15cb1c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:42.516903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] The token cache is being invalidate because project 38fd4e9f0be14117a09e24bc292f11d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:42.517614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:42.519182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3805a745-ac4b-44e9-bafa-844e5a15cb1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:42.528379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 861/1750] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:42 2022] DELETE /identity/v3/projects/38fd4e9f0be14117a09e24bc292f11d7 => generated 0 bytes in 517 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:42.534282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b13e7-52eb-4e80-b7a9-1d07144bfa78 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:42.542953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b13e7-52eb-4e80-b7a9-1d07144bfa78 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:42.542953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4b13e7-52eb-4e80-b7a9-1d07144bfa78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:42.571452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:42.576471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3b4b13e7-52eb-4e80-b7a9-1d07144bfa78 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:42.588636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3b4b13e7-52eb-4e80-b7a9-1d07144bfa78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:42.742442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 890/1751] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:42.743211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee26b33a-df14-43b1-8808-ce3de7a27ee6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:42.744009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee26b33a-df14-43b1-8808-ce3de7a27ee6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:42.744009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee26b33a-df14-43b1-8808-ce3de7a27ee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:42.770962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ee26b33a-df14-43b1-8808-ce3de7a27ee6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:42.774493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ee26b33a-df14-43b1-8808-ce3de7a27ee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:42.794407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b13e7-52eb-4e80-b7a9-1d07144bfa78 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:42.806843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 862/1752] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:42.812835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4b13e7-52eb-4e80-b7a9-1d07144bfa78 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:42.819421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:42.821340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:42.821340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:42.821340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:42.821789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:42.822029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:42.824317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:42.867182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee26b33a-df14-43b1-8808-ce3de7a27ee6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:42.884721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee26b33a-df14-43b1-8808-ce3de7a27ee6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:42.887359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 891/1753] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:42.899209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:42.900675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:42.901010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:42.901293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:42.902212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:42.904996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:43.068662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:43.079580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 863/1754] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:42 2022] POST /identity/v3/projects => generated 385 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:43.092284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:43.100397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b256c23-c3d9-4b11-9f45-5b2224e6b0bd admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:43.225498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:43.226411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:43.226896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:43.227288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:43.228122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:43.230207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:43.345530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 892/1755] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:43 2022] POST /identity/v3/users => generated 293 bytes in 277 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:43.346808 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:43.365907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3380560b-9d3f-4dd7-a1e8-b4cded49840f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:43.384950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:43.387132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:43.388330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:43.389143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:43.390139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:43.392047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:43.413288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 864/1756] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:43 2022] GET /identity/v3/roles => generated 1420 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:43.414377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:43.437977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21b61af5-128d-45f7-985b-d998314f143e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:43.449630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08b4eeb-de64-4d69-86e2-086cc6baa275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:43.450947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08b4eeb-de64-4d69-86e2-086cc6baa275 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:43.451256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08b4eeb-de64-4d69-86e2-086cc6baa275 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:43.451512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08b4eeb-de64-4d69-86e2-086cc6baa275 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:43.452311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08b4eeb-de64-4d69-86e2-086cc6baa275 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:43.454158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08b4eeb-de64-4d69-86e2-086cc6baa275 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:43.503377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 893/1757] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:43 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:18:43.508471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:43.520454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86db730a-fd56-4b03-9aea-dd8b04a857ee admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:43.550386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:43.554649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:43.554959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:43.557303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] PATH_INFO: `/v3/projects/7a4d418f00c74c7ea11f8eb2ba4898c1/users/5f11a17d584847d381223d73e52ec2ad/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:43.595475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] RBAC: Authorizing `identity:create_grant(project_id=7a4d418f00c74c7ea11f8eb2ba4898c1, user_id=5f11a17d584847d381223d73e52ec2ad, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:43.598606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:43.749178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 865/1758] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:43 2022] PUT /identity/v3/projects/7a4d418f00c74c7ea11f8eb2ba4898c1/users/5f11a17d584847d381223d73e52ec2ad/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 241 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:43.765758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ef803f-0355-4442-91b2-bcd8b1310251 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:43.765758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ef803f-0355-4442-91b2-bcd8b1310251 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:43.765758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ef803f-0355-4442-91b2-bcd8b1310251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:43.827028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-86ef803f-0355-4442-91b2-bcd8b1310251 None None] MFA Rules not processed for user `5f11a17d584847d381223d73e52ec2ad`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:43.829560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-86ef803f-0355-4442-91b2-bcd8b1310251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:43.943229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:44.060809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 894/1759] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:43 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 301 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:44.090136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:44.093457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:44.222364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:44.288581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248187f0-20a5-4c3a-ad37-3b562800176b admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:44.302154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5667892b-6a7c-4912-9e2d-f3804f020848 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:44.302867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5667892b-6a7c-4912-9e2d-f3804f020848 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:44.303155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5667892b-6a7c-4912-9e2d-f3804f020848 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:44.303430 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5667892b-6a7c-4912-9e2d-f3804f020848 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:44.324291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5667892b-6a7c-4912-9e2d-f3804f020848 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:44.326599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5667892b-6a7c-4912-9e2d-f3804f020848 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:44.384434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 866/1760] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:44 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:44.950687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66175bb-8b16-40d2-8cc1-1103cd2bb46f None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:44.951389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66175bb-8b16-40d2-8cc1-1103cd2bb46f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:44.951538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66175bb-8b16-40d2-8cc1-1103cd2bb46f None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:44.953502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 895/1761] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:18:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:44.976037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:44.976037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:44.976309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:45.016179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:45.017442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:45.063947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:45.064123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:45.064299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:45.170725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] MFA Rules not processed for user `a6aa2404a2a4422eb6b5bf4496656c7a`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:45.171237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:45.173155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:45.216245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 867/1762] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:18:44 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:45.308225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:45.315940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:45.331899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7096f5f-946b-4d35-8339-b609845f7962 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:45.357014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 896/1763] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:45 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 294 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:45.358034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:45.359458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:45.374325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:45.375545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:45.376121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:45.377352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:45.411294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:45.414574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:45.434868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 868/1764] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:45 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:18:45.529637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff40244-eedf-47b3-ab97-05599f1e7e81 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:45.541605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:45.545788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:45.546992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:45.548016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:45.552490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:45.598992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:45.751726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 897/1765] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:45 2022] POST /identity/v3/projects => generated 405 bytes in 394 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:45.754864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:45.772539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc3da15a-5943-4a88-98ca-60a6fd3d86df service nova] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:45.954025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:45.955101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:45.955533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:45.956147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:45.957205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:45.963902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:45.969262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:45.974080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:45.995092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76685f7f-5a91-4626-a574-f909eb5d17a5 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:46.001169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:46.002065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:46.002622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:46.003091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] PATH_INFO: `/v3/users/de537197766d475a82190f876953e934` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:46.007270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de537197766d475a82190f876953e934)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:46.011094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:46.086489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 869/1766] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:45 2022] POST /identity/v3/users => generated 309 bytes in 332 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:46.091270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:46.101830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79241310-c1c2-4915-807e-892e63589ee1 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:46.107000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:46.107607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:46.107890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:46.108168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:46.108907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:46.110729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:46.139419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 870/1767] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:46.144803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:46.155719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20a9bf4-32e9-4c7a-9753-3e8360434dea admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:46.160640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:46.161687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:46.161993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:46.162348 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] PATH_INFO: `/v3/projects/20c04a41f553455187d978e8dd4ae68a/users/476deb8c6d054d859a155fae9b2fd34c/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:46.181826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20c04a41f553455187d978e8dd4ae68a, user_id=476deb8c6d054d859a155fae9b2fd34c, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:46.184731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:46.269746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de537197766d475a82190f876953e934'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:46.281774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 871/1768] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:46 2022] PUT /identity/v3/projects/20c04a41f553455187d978e8dd4ae68a/users/476deb8c6d054d859a155fae9b2fd34c/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:46.285961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:46.354088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc660c3-6f13-43a8-8141-f1e3aad2fe73 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:46.358824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:46.359517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:46.359797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:46.360068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:46.360791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:46.362391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:46.372296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 872/1769] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:46 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:18:46.376409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:46.388209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77e5ca2-a7c4-4479-9e5c-79ee4680a91a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:46.394280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00898a9b-a692-4ea6-a88c-df125941d514 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:46.395563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00898a9b-a692-4ea6-a88c-df125941d514 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:46.395996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00898a9b-a692-4ea6-a88c-df125941d514 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:46.396956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00898a9b-a692-4ea6-a88c-df125941d514 admin admin] PATH_INFO: `/v3/projects/20c04a41f553455187d978e8dd4ae68a/users/476deb8c6d054d859a155fae9b2fd34c/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:46.398792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00898a9b-a692-4ea6-a88c-df125941d514 admin admin] RBAC: Authorizing `identity:create_grant(project_id=20c04a41f553455187d978e8dd4ae68a, user_id=476deb8c6d054d859a155fae9b2fd34c, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:46.401515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00898a9b-a692-4ea6-a88c-df125941d514 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:46.411651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de537197766d475a82190f876953e934'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:46.413316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 873/1770] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:46 2022] PUT /identity/v3/projects/20c04a41f553455187d978e8dd4ae68a/users/476deb8c6d054d859a155fae9b2fd34c/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:46.423683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de537197766d475a82190f876953e934'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:46.424453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de537197766d475a82190f876953e934'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:46.425075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de537197766d475a82190f876953e934'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:46.435591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:46.458053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 898/1771] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:45 2022] DELETE /identity/v3/users/de537197766d475a82190f876953e934 => generated 0 bytes in 489 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:46.462576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:46.462921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:46.463203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:46.481749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:46.486884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:46.566494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 874/1772] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:46.573891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:46.576014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:46.645614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6447ec0-6756-4754-a7f1-9b9c2bdef1b3 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:46.652609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ebfdc6-ac82-467f-a6a1-18df07e5e9fd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:46.653692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ebfdc6-ac82-467f-a6a1-18df07e5e9fd service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:46.653841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ebfdc6-ac82-467f-a6a1-18df07e5e9fd service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:46.654282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ebfdc6-ac82-467f-a6a1-18df07e5e9fd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:46.668241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ebfdc6-ac82-467f-a6a1-18df07e5e9fd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:46.670055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59ebfdc6-ac82-467f-a6a1-18df07e5e9fd service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:46.707638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 899/1773] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:47.494576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:47.496331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:47.510112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c814fc27-6770-4272-8530-492dca759dc7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:47.516340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:47.518150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:47.518605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:47.519206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] PATH_INFO: `/v3/projects/52b69bf6562e4a25800f13288975e44e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:47.521671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=52b69bf6562e4a25800f13288975e44e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:47.525285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:47.727824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52b69bf6562e4a25800f13288975e44e', 'request_id': 'req-25a23954-c1db-4155-aa45-d49a59741c0c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:47.739843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52b69bf6562e4a25800f13288975e44e', 'request_id': 'req-25a23954-c1db-4155-aa45-d49a59741c0c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:47.850788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52b69bf6562e4a25800f13288975e44e', 'request_id': 'req-25a23954-c1db-4155-aa45-d49a59741c0c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:47.871990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] The token cache is being invalidate because project 52b69bf6562e4a25800f13288975e44e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:47.872530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:47.873569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-25a23954-c1db-4155-aa45-d49a59741c0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:47.875936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 875/1774] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:47 2022] DELETE /identity/v3/projects/52b69bf6562e4a25800f13288975e44e => generated 0 bytes in 382 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:47.881650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa7712d-4dd0-4de2-a790-06437ad07073 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:47.881650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa7712d-4dd0-4de2-a790-06437ad07073 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:47.882429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa7712d-4dd0-4de2-a790-06437ad07073 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:47.908713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2aa7712d-4dd0-4de2-a790-06437ad07073 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:47.924081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2aa7712d-4dd0-4de2-a790-06437ad07073 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:48.089389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 900/1775] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:48.104919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c87cc3b-5520-4166-9712-d96bebc5fcfd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.106696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c87cc3b-5520-4166-9712-d96bebc5fcfd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.107474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c87cc3b-5520-4166-9712-d96bebc5fcfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.163856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4c87cc3b-5520-4166-9712-d96bebc5fcfd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:48.164627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4c87cc3b-5520-4166-9712-d96bebc5fcfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:48.194127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa7712d-4dd0-4de2-a790-06437ad07073 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:48.221732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 876/1776] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:48.226751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa7712d-4dd0-4de2-a790-06437ad07073 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:48.232059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:48.233751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.234215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.234699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.235767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:48.236332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:48.261464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:48.271929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 901/1777] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:48.273045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c87cc3b-5520-4166-9712-d96bebc5fcfd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:48.294446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c87cc3b-5520-4166-9712-d96bebc5fcfd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:48.300052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d4c86f-26e0-4904-8827-f304ba9bc7d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:48.301093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4c86f-26e0-4904-8827-f304ba9bc7d8 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.301445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4c86f-26e0-4904-8827-f304ba9bc7d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.301785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d4c86f-26e0-4904-8827-f304ba9bc7d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.302576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d4c86f-26e0-4904-8827-f304ba9bc7d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:48.304875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d4c86f-26e0-4904-8827-f304ba9bc7d8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:48.409408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 877/1778] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:48 2022] POST /identity/v3/projects => generated 375 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:48.413762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:48.440699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e82c66c-cb2d-4d43-9c92-8abdeaffb8c4 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:48.502687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:48.561664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:48.562364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.562649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.563067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.563807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:48.565828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:48.587419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9150b1-778b-4677-8b36-7dc6de4e3b97 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.587844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9150b1-778b-4677-8b36-7dc6de4e3b97 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.588133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9150b1-778b-4677-8b36-7dc6de4e3b97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.644055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-0b9150b1-778b-4677-8b36-7dc6de4e3b97 None None] MFA Rules not processed for user `476deb8c6d054d859a155fae9b2fd34c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:48.645645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0b9150b1-778b-4677-8b36-7dc6de4e3b97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:48.673935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 902/1779] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:48 2022] POST /identity/v3/users => generated 288 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:48.675292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:48.687783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7b6930-6d7c-4467-b7c5-15ad135e5f16 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:48.697206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:48.697900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.698197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.698476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.700870 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:48.702731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:48.725139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 903/1780] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:48 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:48.733549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:48.754354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e4cb91-014a-4529-8765-5f3d5f5e6b09 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:48.759100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:48.759775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.760061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.760339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.761070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:48.763931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:48.775391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 904/1781] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:48 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:48.780875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:48.783283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 878/1782] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:48 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:48.797258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99025fef-db90-46c3-bb08-42ff6b4c7d2a admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:48.802374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:48.805209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.805209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.805209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] PATH_INFO: `/v3/projects/75d81fcbf8bf44449966ccb19f79665b/users/2fd51e56a8c54ce3a4d00c8213390773/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.819466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75d81fcbf8bf44449966ccb19f79665b, user_id=2fd51e56a8c54ce3a4d00c8213390773, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:48.823359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:48.960007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 905/1783] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:48 2022] PUT /identity/v3/projects/75d81fcbf8bf44449966ccb19f79665b/users/2fd51e56a8c54ce3a4d00c8213390773/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:48.970224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d43f3-3f1f-4ded-97f4-114643a13efd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:48.970512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d43f3-3f1f-4ded-97f4-114643a13efd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:48.970816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d13d43f3-3f1f-4ded-97f4-114643a13efd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:48.997155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d13d43f3-3f1f-4ded-97f4-114643a13efd None None] MFA Rules not processed for user `2fd51e56a8c54ce3a4d00c8213390773`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:48.998448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d13d43f3-3f1f-4ded-97f4-114643a13efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:49.073752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 879/1784] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:48 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:49.082981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:49.084763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:49.187175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379b0906-88fe-4968-af16-47799ef64be3 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:49.204120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db1e7a6-9027-4535-822b-42d735508bb4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:49.204805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1e7a6-9027-4535-822b-42d735508bb4 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:49.205096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1e7a6-9027-4535-822b-42d735508bb4 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:49.205380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db1e7a6-9027-4535-822b-42d735508bb4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:49.216078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db1e7a6-9027-4535-822b-42d735508bb4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:49.217595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db1e7a6-9027-4535-822b-42d735508bb4 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:49.229703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 906/1785] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:49 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:49.482606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13d43f3-3f1f-4ded-97f4-114643a13efd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:49.556948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d13d43f3-3f1f-4ded-97f4-114643a13efd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:49.562243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:49.563239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:49.563522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:49.563798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] PATH_INFO: `/v3/users/2fd51e56a8c54ce3a4d00c8213390773` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:49.565019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2fd51e56a8c54ce3a4d00c8213390773)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:49.567025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:49.591924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f05fd0d-8881-4f7d-b7b6-5d2ce873b979 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:49.592174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f05fd0d-8881-4f7d-b7b6-5d2ce873b979 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:49.592414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f05fd0d-8881-4f7d-b7b6-5d2ce873b979 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:49.593918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 907/1786] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:18:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 14 12:18:49.631027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:49.631531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:49.631886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:49.650633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:49.651890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:49.737300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 908/1787] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:18:49 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:49.769179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:49.771208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:49.806816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b169522-3e89-41b7-9ec7-ce042124cf66 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:49.816257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa17230-b36b-4983-989c-3774ed4f33df service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:49.816925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa17230-b36b-4983-989c-3774ed4f33df service placement] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:49.817204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa17230-b36b-4983-989c-3774ed4f33df service placement] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:49.817619 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa17230-b36b-4983-989c-3774ed4f33df service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:49.827901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa17230-b36b-4983-989c-3774ed4f33df service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:49.829653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa17230-b36b-4983-989c-3774ed4f33df service placement] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:49.843632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 909/1788] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:49 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:49.903915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2fd51e56a8c54ce3a4d00c8213390773'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:50.050488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2fd51e56a8c54ce3a4d00c8213390773'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:50.051133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2fd51e56a8c54ce3a4d00c8213390773'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:50.059073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2fd51e56a8c54ce3a4d00c8213390773'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:50.059740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2fd51e56a8c54ce3a4d00c8213390773'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:50.080395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 880/1789] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:49 2022] DELETE /identity/v3/users/2fd51e56a8c54ce3a4d00c8213390773 => generated 0 bytes in 598 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:50.084941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39127585-6fc6-4a50-9046-09a408f17216 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:50.085419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39127585-6fc6-4a50-9046-09a408f17216 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:50.085614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39127585-6fc6-4a50-9046-09a408f17216 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:50.114193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-39127585-6fc6-4a50-9046-09a408f17216 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:50.115705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-39127585-6fc6-4a50-9046-09a408f17216 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:50.215683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 910/1790] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:50.224542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:50.226072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:50.287872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2932cda4-1840-4528-88b7-8906b0d2294a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:50.292152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9dbe6a1-4ef9-4bef-9535-c6d9f0ebc725 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:50.292786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dbe6a1-4ef9-4bef-9535-c6d9f0ebc725 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:50.294194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dbe6a1-4ef9-4bef-9535-c6d9f0ebc725 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:50.294529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9dbe6a1-4ef9-4bef-9535-c6d9f0ebc725 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:50.307975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dbe6a1-4ef9-4bef-9535-c6d9f0ebc725 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:50.308507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9dbe6a1-4ef9-4bef-9535-c6d9f0ebc725 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:50.322262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 881/1791] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:50.632241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2814055-a23f-4e77-b2f0-ccc203992ee5 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:50.632576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2814055-a23f-4e77-b2f0-ccc203992ee5 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:50.634626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2814055-a23f-4e77-b2f0-ccc203992ee5 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:50.634626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 911/1792] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:18:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:50.654563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:50.654563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:50.654563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:50.677398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] MFA Rules not processed for user `a9c50ae1ba494c20b6133a96dfba21b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:50.678623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:50.729443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:50.752491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 882/1793] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:18:50 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:50.763526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2814055-a23f-4e77-b2f0-ccc203992ee5 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:50.765254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d2814055-a23f-4e77-b2f0-ccc203992ee5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:50.788764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2814055-a23f-4e77-b2f0-ccc203992ee5 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:50.794409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:50.796829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:50.804999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:50.806480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:50.806480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:50.806480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:50.812865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5139a3ba-3aa2-4b28-beb4-9a45b7a26a9f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:50.819191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:50.820610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:50.821120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:50.821553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] PATH_INFO: `/v3/users/a6aa2404a2a4422eb6b5bf4496656c7a` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:50.823079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:50.823758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6aa2404a2a4422eb6b5bf4496656c7a)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:50.824831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:50.826138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:50.837818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 912/1794] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:50 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:18:50.882101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:50.883598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:50.899901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a106578-bbba-466a-b3a4-7422843e28da service placement] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:50.905721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:50.906943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:50.907390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:50.907749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] PATH_INFO: `/v3/projects/75d81fcbf8bf44449966ccb19f79665b` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:50.909133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75d81fcbf8bf44449966ccb19f79665b)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:50.911335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.125297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75d81fcbf8bf44449966ccb19f79665b', 'request_id': 'req-be2dbc06-ddbe-414a-831c-2dd4648e73a0'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.133109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6aa2404a2a4422eb6b5bf4496656c7a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.222473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:51.250556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75d81fcbf8bf44449966ccb19f79665b', 'request_id': 'req-be2dbc06-ddbe-414a-831c-2dd4648e73a0'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.257954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75d81fcbf8bf44449966ccb19f79665b', 'request_id': 'req-be2dbc06-ddbe-414a-831c-2dd4648e73a0'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.268937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] The token cache is being invalidate because project 75d81fcbf8bf44449966ccb19f79665b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:51.269229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.269900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-be2dbc06-ddbe-414a-831c-2dd4648e73a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.271556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 913/1795] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:50 2022] DELETE /identity/v3/projects/75d81fcbf8bf44449966ccb19f79665b => generated 0 bytes in 390 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:51.276238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865836a2-6d66-445f-8356-25fb21f5ba2b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.276531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865836a2-6d66-445f-8356-25fb21f5ba2b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.276817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865836a2-6d66-445f-8356-25fb21f5ba2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.296255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-865836a2-6d66-445f-8356-25fb21f5ba2b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:51.297458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-865836a2-6d66-445f-8356-25fb21f5ba2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:51.325581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6aa2404a2a4422eb6b5bf4496656c7a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.326313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6aa2404a2a4422eb6b5bf4496656c7a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.333870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6aa2404a2a4422eb6b5bf4496656c7a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.334797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fc3be39f-c26c-4c38-9f0a-1f25db9f9cec admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6aa2404a2a4422eb6b5bf4496656c7a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.354075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 883/1796] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:50 2022] DELETE /identity/v3/users/a6aa2404a2a4422eb6b5bf4496656c7a => generated 0 bytes in 561 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:51.356877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 914/1797] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:51.365739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b086f5-10d9-4ce8-8ab0-6b818d5576b4 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.366035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b086f5-10d9-4ce8-8ab0-6b818d5576b4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.366319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83b086f5-10d9-4ce8-8ab0-6b818d5576b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.384759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-83b086f5-10d9-4ce8-8ab0-6b818d5576b4 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:51.386015 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-83b086f5-10d9-4ce8-8ab0-6b818d5576b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:51.454539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 884/1798] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:51.458650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865836a2-6d66-445f-8356-25fb21f5ba2b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.469337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865836a2-6d66-445f-8356-25fb21f5ba2b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.478357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.478357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.478357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.478357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.478651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:51.478651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.478651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.490257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 915/1799] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:51.494502 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b086f5-10d9-4ce8-8ab0-6b818d5576b4 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.503838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83b086f5-10d9-4ce8-8ab0-6b818d5576b4 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.508477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.509141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.509420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.512695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.512695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.513021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.600717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.602717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:51.613681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 885/1800] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:51 2022] POST /identity/v3/projects => generated 371 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:51.618851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.621021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea17962-d10c-4991-b9ef-d713047acba4 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.679025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.679905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.680184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.680460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] PATH_INFO: `/v3/projects/7cdf723129954ef4b166727c84dd9d57` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.681654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7cdf723129954ef4b166727c84dd9d57)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.682984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cf8dd3-0e28-4491-8c8a-c0e50947d4e0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.683777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.688729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.689446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.689734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.690013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.690757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.692758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.804127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 886/1801] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:51 2022] POST /identity/v3/users => generated 286 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:51.805351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.813585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7cdf723129954ef4b166727c84dd9d57', 'request_id': 'req-107ac961-44df-4dbe-acfe-62bc932ba828'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.816499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40d9a4a-8dad-4072-ad0a-3f4a391e107b admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.875566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.876242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.876532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.876865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.877603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.879287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.881020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7cdf723129954ef4b166727c84dd9d57', 'request_id': 'req-107ac961-44df-4dbe-acfe-62bc932ba828'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.888917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7cdf723129954ef4b166727c84dd9d57', 'request_id': 'req-107ac961-44df-4dbe-acfe-62bc932ba828'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.891798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 887/1802] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:51 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:18:51.895957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.905923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f95a2d-682f-4336-b3cc-e82a7a50868c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.906073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] The token cache is being invalidate because project 7cdf723129954ef4b166727c84dd9d57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:51.906351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.907038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:51.910899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 916/1803] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:51 2022] DELETE /identity/v3/projects/7cdf723129954ef4b166727c84dd9d57 => generated 0 bytes in 309 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:51.912512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.913303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.913912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:51.914063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.914448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.914846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.915722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.917307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.926873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107ac961-44df-4dbe-acfe-62bc932ba828 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.928210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 888/1804] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:51.932360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.932535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:51.933241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.933528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.933803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] PATH_INFO: `/v3/users/476deb8c6d054d859a155fae9b2fd34c` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.934877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:51.935816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=476deb8c6d054d859a155fae9b2fd34c)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.937764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:51.947457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1b2246-e111-4c32-a545-a059c1edc458 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:51.952994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b4809c-d89b-45fc-836f-dd9655c7b0d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:51.953928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4809c-d89b-45fc-836f-dd9655c7b0d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:51.954211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4809c-d89b-45fc-836f-dd9655c7b0d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:51.954490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b4809c-d89b-45fc-836f-dd9655c7b0d1 admin admin] PATH_INFO: `/v3/projects/19abd680a3ef4bd2bf3a43329bfac025/users/cd9eea52c8394dcb950be4ca65604d78/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:51.969458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b4809c-d89b-45fc-836f-dd9655c7b0d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=19abd680a3ef4bd2bf3a43329bfac025, user_id=cd9eea52c8394dcb950be4ca65604d78, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:51.972158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b4809c-d89b-45fc-836f-dd9655c7b0d1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:52.084531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 889/1805] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:51 2022] PUT /identity/v3/projects/19abd680a3ef4bd2bf3a43329bfac025/users/cd9eea52c8394dcb950be4ca65604d78/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:52.094850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d203688c-b2b6-4a2a-8d24-22de9aa81d7d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:52.095072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d203688c-b2b6-4a2a-8d24-22de9aa81d7d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:52.095347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d203688c-b2b6-4a2a-8d24-22de9aa81d7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:52.122253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d203688c-b2b6-4a2a-8d24-22de9aa81d7d None None] MFA Rules not processed for user `cd9eea52c8394dcb950be4ca65604d78`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:52.123632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d203688c-b2b6-4a2a-8d24-22de9aa81d7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:52.146348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '476deb8c6d054d859a155fae9b2fd34c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:52.198448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 890/1806] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:52 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:52.205145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d203688c-b2b6-4a2a-8d24-22de9aa81d7d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:52.239802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '476deb8c6d054d859a155fae9b2fd34c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:52.254075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '476deb8c6d054d859a155fae9b2fd34c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:52.254945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '476deb8c6d054d859a155fae9b2fd34c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:52.255770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '476deb8c6d054d859a155fae9b2fd34c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:52.290815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 917/1807] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:51 2022] DELETE /identity/v3/users/476deb8c6d054d859a155fae9b2fd34c => generated 0 bytes in 376 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:52.360679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d203688c-b2b6-4a2a-8d24-22de9aa81d7d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:52.433728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:52.435316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:52.435596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:52.435874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:52.436577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:52.438604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:52.546631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 891/1808] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:52 2022] POST /identity/v3/projects => generated 373 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:52.549794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:52.552010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:52.584568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-760572aa-cecb-4399-bdb8-5e14aa195bf5 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:52.646455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:52.647238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:52.647524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:52.647804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:52.648661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:52.651270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:52.686526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:52.792115 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 918/1809] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:52 2022] POST /identity/v3/users => generated 293 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:52.797196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:52.814158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ee0b3c-db70-42cb-9a6a-0e6788f92d77 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:52.820251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:52.821440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:52.821988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:52.823163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:52.824170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:52.827807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:52.841965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 892/1810] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:52 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:18:52.846344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:52.858129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f270c51-fede-49f1-84c3-7db81bcbe2bb admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:52.865163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:52.868898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:52.868898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:52.868898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] PATH_INFO: `/v3/projects/4aadaa088fae43058bcefd9a0eb5c80a/users/545620adb5c041ee8519d665f9905cc8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:52.886086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] RBAC: Authorizing `identity:create_grant(project_id=4aadaa088fae43058bcefd9a0eb5c80a, user_id=545620adb5c041ee8519d665f9905cc8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:52.890161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:53.029541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 919/1811] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:52 2022] PUT /identity/v3/projects/4aadaa088fae43058bcefd9a0eb5c80a/users/545620adb5c041ee8519d665f9905cc8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:53.030575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:53.043437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d1f4bc-7d91-4dd6-83f5-4d7264d0c8dc admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:53.129807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:53.132427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:53.138720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:53.142602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.143481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.146343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.146343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:53.147840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:53.164423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84ab7b23-4988-4cba-89ff-d435eceffe0e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:53.169388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 893/1812] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:53 2022] GET /identity/v3/roles => generated 1420 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:18:53.173506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:53.206421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:53.207789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.208238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.208638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] PATH_INFO: `/v3/projects/20c04a41f553455187d978e8dd4ae68a` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.210786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] RBAC: Authorizing `identity:delete_project(project_id=20c04a41f553455187d978e8dd4ae68a)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:53.217605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:53.229276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad897d72-7786-4204-af56-04699e2ec8a9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:53.263630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0074d39-6c99-486b-bb2c-108b9e3b2592 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:53.264609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0074d39-6c99-486b-bb2c-108b9e3b2592 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.264949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0074d39-6c99-486b-bb2c-108b9e3b2592 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.265235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0074d39-6c99-486b-bb2c-108b9e3b2592 admin admin] PATH_INFO: `/v3/projects/4aadaa088fae43058bcefd9a0eb5c80a/users/545620adb5c041ee8519d665f9905cc8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.267608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0074d39-6c99-486b-bb2c-108b9e3b2592 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4aadaa088fae43058bcefd9a0eb5c80a, user_id=545620adb5c041ee8519d665f9905cc8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:53.270861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0074d39-6c99-486b-bb2c-108b9e3b2592 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:53.281082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:53.282951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 894/1813] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:53 2022] PUT /identity/v3/projects/4aadaa088fae43058bcefd9a0eb5c80a/users/545620adb5c041ee8519d665f9905cc8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:53.300034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.300330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.300782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.333826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] MFA Rules not processed for user `545620adb5c041ee8519d665f9905cc8`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:53.348564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:53.383630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '20c04a41f553455187d978e8dd4ae68a', 'request_id': 'req-8a85fb8a-e3f7-43a9-9371-11657898f8be'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:53.530683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '20c04a41f553455187d978e8dd4ae68a', 'request_id': 'req-8a85fb8a-e3f7-43a9-9371-11657898f8be'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:53.548656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '20c04a41f553455187d978e8dd4ae68a', 'request_id': 'req-8a85fb8a-e3f7-43a9-9371-11657898f8be'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:53.578193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 895/1814] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:53 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 279 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:53.609217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] The token cache is being invalidate because project 20c04a41f553455187d978e8dd4ae68a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:53.609217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:53.623358 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:53.625019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:53.626826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-8a85fb8a-e3f7-43a9-9371-11657898f8be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:53.642197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 920/1815] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:53 2022] DELETE /identity/v3/projects/20c04a41f553455187d978e8dd4ae68a => generated 0 bytes in 513 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:53.647429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a0d39-9ae1-4a6f-8943-1b89d0d859b2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.648363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a0d39-9ae1-4a6f-8943-1b89d0d859b2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.655268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6a0d39-9ae1-4a6f-8943-1b89d0d859b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.701636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3f6a0d39-9ae1-4a6f-8943-1b89d0d859b2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:53.702953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3f6a0d39-9ae1-4a6f-8943-1b89d0d859b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:53.812575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d705165c-b776-40e3-99a1-a48ab772ea72 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:53.829149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:53.829920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.830273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.830669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.835596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:53.861134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:53.863463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:53.877987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 921/1816] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:53.886554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf933c7-de69-4f6c-b4fc-d446d0f7107b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.886878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf933c7-de69-4f6c-b4fc-d446d0f7107b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.890435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 896/1817] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:53 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 266 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:53.892145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf933c7-de69-4f6c-b4fc-d446d0f7107b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.924594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ecf933c7-de69-4f6c-b4fc-d446d0f7107b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:53.925282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ecf933c7-de69-4f6c-b4fc-d446d0f7107b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:53.934086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 922/1818] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:53.938904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:53.951069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d49acd65-07bc-42d3-a909-ef733e0218cf service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:53.964719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:53.964719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:53.964719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:53.964719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:53.965044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:53.965044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:53.965044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:53.984153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf933c7-de69-4f6c-b4fc-d446d0f7107b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:53.992805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 897/1819] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:53.999765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf933c7-de69-4f6c-b4fc-d446d0f7107b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:54.007559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a25cdaa-7453-4a0e-9371-d68cfa444dde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:54.008391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25cdaa-7453-4a0e-9371-d68cfa444dde admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.008974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25cdaa-7453-4a0e-9371-d68cfa444dde admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.009252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a25cdaa-7453-4a0e-9371-d68cfa444dde admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.009974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a25cdaa-7453-4a0e-9371-d68cfa444dde admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:54.012864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a25cdaa-7453-4a0e-9371-d68cfa444dde admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:54.239710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:54.247799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 923/1820] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:53 2022] POST /identity/v3/projects => generated 381 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:54.254825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079c1589-b257-495a-a034-cadcb5bb302c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:54.399138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399ee93c-c426-48c5-8ea2-8a8d13fef9bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:54.400554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ee93c-c426-48c5-8ea2-8a8d13fef9bd admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.400920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ee93c-c426-48c5-8ea2-8a8d13fef9bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.406637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399ee93c-c426-48c5-8ea2-8a8d13fef9bd admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.407592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399ee93c-c426-48c5-8ea2-8a8d13fef9bd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:54.410270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-399ee93c-c426-48c5-8ea2-8a8d13fef9bd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:54.414913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:54.510158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0696052-56fb-40e7-bab6-cba1dd8522bb None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.511380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0696052-56fb-40e7-bab6-cba1dd8522bb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.513108 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0696052-56fb-40e7-bab6-cba1dd8522bb None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.522040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 924/1821] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:18:54 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:18:54.522258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 898/1822] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:54 2022] POST /identity/v3/users => generated 291 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:54.526859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0696052-56fb-40e7-bab6-cba1dd8522bb None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:54.555466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0696052-56fb-40e7-bab6-cba1dd8522bb None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:54.565060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.565385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.565618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.567670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:54.568353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.569399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.569399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.569673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:54.581559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:54.620531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:54.622334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:54.632809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 925/1823] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:54 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:54.634341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:54.722615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c3e60bd-55ed-4db1-84f3-ca847a7d0260 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:54.734264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:54.735335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.735622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.735914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.736646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:54.748348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:54.778415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 926/1824] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:54 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:54.779957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:54.804620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42612680-d3c3-4098-9da9-fc847f7ddeaa admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:54.812283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:54.822488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.822488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.822488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] PATH_INFO: `/v3/projects/7efe566f45cd4e3ea84901d0439373c2/users/37489203c7a94fe0b55cd1a57a72da92/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.851649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=7efe566f45cd4e3ea84901d0439373c2, user_id=37489203c7a94fe0b55cd1a57a72da92, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:54.854852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:54.863896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 899/1825] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:18:54 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 299 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:54.864465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:54.866122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:54.888111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e520a07-98d0-4eff-bc68-b184354d327c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:54.893783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:54.901447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8696500f-8f32-4a09-89da-7f86a7419c26 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:54.902126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8696500f-8f32-4a09-89da-7f86a7419c26 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.902414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8696500f-8f32-4a09-89da-7f86a7419c26 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.903248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8696500f-8f32-4a09-89da-7f86a7419c26 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:54.913661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8696500f-8f32-4a09-89da-7f86a7419c26 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:54.915611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8696500f-8f32-4a09-89da-7f86a7419c26 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:54.942941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 900/1826] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:54 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 79 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:54.952572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 927/1827] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:54 2022] PUT /identity/v3/projects/7efe566f45cd4e3ea84901d0439373c2/users/37489203c7a94fe0b55cd1a57a72da92/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:54.964488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b4a42a-5d05-4efe-b886-16c7569605ff None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:54.964778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b4a42a-5d05-4efe-b886-16c7569605ff None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:54.966029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b4a42a-5d05-4efe-b886-16c7569605ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.001584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-77b4a42a-5d05-4efe-b886-16c7569605ff None None] MFA Rules not processed for user `37489203c7a94fe0b55cd1a57a72da92`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:55.005454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-77b4a42a-5d05-4efe-b886-16c7569605ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:55.091299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 901/1828] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:54 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:55.095961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.171623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e561334-870f-4180-a27f-b09b7ef33e3c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.177205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.177924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.178370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.180521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.180521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:55.181842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:55.306734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:55.311756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 928/1829] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:55 2022] POST /identity/v3/projects => generated 379 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:55.319585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b4a42a-5d05-4efe-b886-16c7569605ff None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.339464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b4a42a-5d05-4efe-b886-16c7569605ff None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.414237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.415108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.415542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.415688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.416454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:55.418675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:55.457474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.459245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:55.472880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b45f6d7-7b66-4e20-84e8-8e6c2e958b56 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.477800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.478735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.479265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.481180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] PATH_INFO: `/v3/users/cd9eea52c8394dcb950be4ca65604d78` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.481180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] RBAC: Authorizing `identity:delete_user(user_id=cd9eea52c8394dcb950be4ca65604d78)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:55.483105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:55.514862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 902/1830] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:55 2022] POST /identity/v3/users => generated 296 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:55.518679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.528801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3e0f73e-4f7f-4550-b76d-7f265d7a1223 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.533519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.534247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.534524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.534834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.535560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:55.537144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:55.547602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 903/1831] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:55 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:18:55.552725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.562491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59012ff3-78f1-451b-9c98-6e8fce5b6c7f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.567296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.568148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.568473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.568744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] PATH_INFO: `/v3/projects/e0baae198b7749d199d63a41f25f828e/users/d8c167cd14d44ab9b9883e9b1c6f24d2/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.585365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0baae198b7749d199d63a41f25f828e, user_id=d8c167cd14d44ab9b9883e9b1c6f24d2, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:55.588103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:55.657929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cd9eea52c8394dcb950be4ca65604d78'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:55.707977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 904/1832] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:55 2022] PUT /identity/v3/projects/e0baae198b7749d199d63a41f25f828e/users/d8c167cd14d44ab9b9883e9b1c6f24d2/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:55.713986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.730587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cd9eea52c8394dcb950be4ca65604d78'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:55.739982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cd9eea52c8394dcb950be4ca65604d78'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:55.740721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cd9eea52c8394dcb950be4ca65604d78'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:55.741480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0c4ccaf-1004-495f-8682-521254d7173b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cd9eea52c8394dcb950be4ca65604d78'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:55.759449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:55.775280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 929/1833] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:55 2022] DELETE /identity/v3/users/cd9eea52c8394dcb950be4ca65604d78 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:55.775280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.775563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.776044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.791356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62bbaf0-623c-4959-bd2c-e086016cfb36 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.798724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:55.798724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:55.859567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.860359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.860695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.860946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.861687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:55.863440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:55.868910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 930/1834] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:55.878347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 905/1835] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:55 2022] GET /identity/v3/roles => generated 1420 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:55.879326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.879437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:55.880884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:55.880998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:55.894563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93a9dca-4fea-4079-8fce-ffefc5bc6408 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.899523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dfa0eb2-5348-4cb0-b14a-5c6392425eac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.900479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa0eb2-5348-4cb0-b14a-5c6392425eac admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.900893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa0eb2-5348-4cb0-b14a-5c6392425eac admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.901574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dfa0eb2-5348-4cb0-b14a-5c6392425eac admin admin] PATH_INFO: `/v3/projects/e0baae198b7749d199d63a41f25f828e/users/d8c167cd14d44ab9b9883e9b1c6f24d2/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.904148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfa0eb2-5348-4cb0-b14a-5c6392425eac admin admin] RBAC: Authorizing `identity:create_grant(project_id=e0baae198b7749d199d63a41f25f828e, user_id=d8c167cd14d44ab9b9883e9b1c6f24d2, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:55.906419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dfa0eb2-5348-4cb0-b14a-5c6392425eac admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:55.919054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 931/1836] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:55 2022] PUT /identity/v3/projects/e0baae198b7749d199d63a41f25f828e/users/d8c167cd14d44ab9b9883e9b1c6f24d2/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:55.929657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.930290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.930290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:55.943365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43bb7cdd-728d-47eb-9b85-3b2e7d5e462e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:55.959636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] MFA Rules not processed for user `d8c167cd14d44ab9b9883e9b1c6f24d2`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:55.960862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:55.992498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:55.993210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:55.993501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:55.993777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:56.038872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 932/1837] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:55 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:18:56.053874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:56.056815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:56.065087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:56.066589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:56.076730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 906/1838] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:56.115842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3364eb2c-5306-45d7-8430-a18b786519ba None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:56.120332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:56.121052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:56.121458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:56.122185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:56.133435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:56.135296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:56.146195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 933/1839] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:56 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:18:56.412541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:56.543992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:56.553314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:56.555129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:56.576486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e81754a2-2cd3-4814-9e02-83f135aed554 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:56.581636 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:56.582663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:56.583041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:56.583391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] PATH_INFO: `/v3/projects/19abd680a3ef4bd2bf3a43329bfac025` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:56.584759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] RBAC: Authorizing `identity:delete_project(project_id=19abd680a3ef4bd2bf3a43329bfac025)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:56.586808 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:56.734104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '19abd680a3ef4bd2bf3a43329bfac025', 'request_id': 'req-e1b4a504-350d-4b23-96d0-900370df5922'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:56.739907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '19abd680a3ef4bd2bf3a43329bfac025', 'request_id': 'req-e1b4a504-350d-4b23-96d0-900370df5922'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:56.811200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '19abd680a3ef4bd2bf3a43329bfac025', 'request_id': 'req-e1b4a504-350d-4b23-96d0-900370df5922'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:56.824267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] The token cache is being invalidate because project 19abd680a3ef4bd2bf3a43329bfac025 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:56.825227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:56.825899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:56.827522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 907/1840] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:56 2022] DELETE /identity/v3/projects/19abd680a3ef4bd2bf3a43329bfac025 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:56.831801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:56.844174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e037c2-50d1-429e-950c-0a7a333860e6 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:56.908830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:56.909747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:56.910085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:56.910412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] PATH_INFO: `/v3/users/545620adb5c041ee8519d665f9905cc8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:56.911780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] RBAC: Authorizing `identity:delete_user(user_id=545620adb5c041ee8519d665f9905cc8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:56.913815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:57.122301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '545620adb5c041ee8519d665f9905cc8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:57.285622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '545620adb5c041ee8519d665f9905cc8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:57.296756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '545620adb5c041ee8519d665f9905cc8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:57.297559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '545620adb5c041ee8519d665f9905cc8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:57.298621 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '545620adb5c041ee8519d665f9905cc8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:57.332612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 934/1841] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:56 2022] DELETE /identity/v3/users/545620adb5c041ee8519d665f9905cc8 => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:57.384717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:57.387522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:57.462758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b4a504-350d-4b23-96d0-900370df5922 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:57.522781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8c167cd14d44ab9b9883e9b1c6f24d2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e0baae198b7749d199d63a41f25f828e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:57.523705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:57.523999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:57.524280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:57.525034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:57.525378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:57.527120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:57.540697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 908/1842] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:57.545760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:57.548314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:57.563515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25c7f19c-86ad-4391-b52b-89926981554a admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:57.568353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8c167cd14d44ab9b9883e9b1c6f24d2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e0baae198b7749d199d63a41f25f828e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:57.569612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:57.570438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:57.570438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:57.571269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:57.573437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:57.713716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 935/1843] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:57 2022] POST /identity/v3/projects => generated 355 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:57.715593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:57.725682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:57.731298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:57.732433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae6368d-c9fd-45d1-874c-fef9ce6cef2b tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:57.734597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:57.808276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8c167cd14d44ab9b9883e9b1c6f24d2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e0baae198b7749d199d63a41f25f828e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:57.810425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:57.810425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:57.810425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:57.810988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:57.811308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:57.813121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:57.822212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535d28ca-af10-457f-8a10-232c71f4b949 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:57.829784 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 909/1844] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 115 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:57.831177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:57.831968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:57.833796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:57.834292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:57.834636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] PATH_INFO: `/v3/users/5f11a17d584847d381223d73e52ec2ad` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:57.836379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5f11a17d584847d381223d73e52ec2ad)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:57.838901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:57.864732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:57.898012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26b54850-1318-40ef-b5f9-f0804eb07fb2 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:57.903728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:57.904781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:57.905072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:57.905494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] PATH_INFO: `/v3/projects/4aadaa088fae43058bcefd9a0eb5c80a` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:57.906918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4aadaa088fae43058bcefd9a0eb5c80a)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:57.908974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:58.078508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4aadaa088fae43058bcefd9a0eb5c80a', 'request_id': 'req-446b7a57-5424-4d28-9449-8097ad518257'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.084907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4aadaa088fae43058bcefd9a0eb5c80a', 'request_id': 'req-446b7a57-5424-4d28-9449-8097ad518257'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.085578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5f11a17d584847d381223d73e52ec2ad'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.126074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:58.142614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4aadaa088fae43058bcefd9a0eb5c80a', 'request_id': 'req-446b7a57-5424-4d28-9449-8097ad518257'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.156473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] The token cache is being invalidate because project 4aadaa088fae43058bcefd9a0eb5c80a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:58.156767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.157437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.159303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 910/1845] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:57 2022] DELETE /identity/v3/projects/4aadaa088fae43058bcefd9a0eb5c80a => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:58.160304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:58.161769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:58.200062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5f11a17d584847d381223d73e52ec2ad'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.208778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5f11a17d584847d381223d73e52ec2ad'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.210364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5f11a17d584847d381223d73e52ec2ad'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.211261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-68fce416-e4f7-425a-b692-61e0f094e92b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5f11a17d584847d381223d73e52ec2ad'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.236105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 936/1846] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:18:57 2022] DELETE /identity/v3/users/5f11a17d584847d381223d73e52ec2ad => generated 0 bytes in 511 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:58.237676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f408e7f0-8692-475e-b9f3-15874921e917 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.238191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f408e7f0-8692-475e-b9f3-15874921e917 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.238561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f408e7f0-8692-475e-b9f3-15874921e917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.242295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-446b7a57-5424-4d28-9449-8097ad518257 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:58.261609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f408e7f0-8692-475e-b9f3-15874921e917 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:58.263726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f408e7f0-8692-475e-b9f3-15874921e917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:58.313651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd8c167cd14d44ab9b9883e9b1c6f24d2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'e0baae198b7749d199d63a41f25f828e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:58.314594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.314966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.315552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] PATH_INFO: `/v3/projects/dce128fd9bb3492784c18ea4027d3bbc` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.316591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorizing `identity:delete_project(project_id=dce128fd9bb3492784c18ea4027d3bbc)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:58.318584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:58.343995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 937/1847] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:58.353664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def7f07c-a4a7-44b3-a80c-4da29735c7e7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.353664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def7f07c-a4a7-44b3-a80c-4da29735c7e7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.353664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def7f07c-a4a7-44b3-a80c-4da29735c7e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.370515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-def7f07c-a4a7-44b3-a80c-4da29735c7e7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:58.371899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-def7f07c-a4a7-44b3-a80c-4da29735c7e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:58.380010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 938/1848] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:58.384701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.384701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.385400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.406412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:58.407771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:58.412069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dce128fd9bb3492784c18ea4027d3bbc', 'request_id': 'req-8e10e47f-d021-4a04-81fc-a0b79b309518'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.417834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dce128fd9bb3492784c18ea4027d3bbc', 'request_id': 'req-8e10e47f-d021-4a04-81fc-a0b79b309518'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.473414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 939/1849] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:58.474257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:58.475600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:58.491040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dce128fd9bb3492784c18ea4027d3bbc', 'request_id': 'req-8e10e47f-d021-4a04-81fc-a0b79b309518'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.507065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] The token cache is being invalidate because project dce128fd9bb3492784c18ea4027d3bbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:18:58.508067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.509035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:18:58.510700 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 911/1850] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:58 2022] DELETE /identity/v3/projects/dce128fd9bb3492784c18ea4027d3bbc => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:18:58.515411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:58.515411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:58.549882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e10e47f-d021-4a04-81fc-a0b79b309518 tempest-VolumeQuotaClassesTest-561781953 tempest-VolumeQuotaClassesTest-561781953-project-admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:58.562673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3ea740-4099-40c9-80f5-e25012b969bf None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:58.563934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:58.564801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.565092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.565439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.566107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:18:58.566806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:58.568101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:58.578790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:58.579449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.579763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.580038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.582355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:58.601197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:58.602171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 912/1851] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:18:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:18:58.603327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:58.607094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:58.614441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 940/1852] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:18:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:18:58.624753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b13599-6ea1-4e2b-b7af-a53c623ad0dd admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:58.630187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:58.630961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.631310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.631649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.635560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:58.635560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:58.705243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 913/1853] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:58 2022] POST /identity/v3/projects => generated 387 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:18:58.709332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:58.721028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9b668c-9489-4c0f-ae6a-0a1b9ba7d3a4 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:58.809520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:58.810346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.810566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.810969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.811858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:58.813824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:58.926711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:58.930872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 941/1854] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:18:58 2022] POST /identity/v3/users => generated 294 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:18:58.940277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae446d1-0100-49d7-86b8-205b8df9ce26 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:58.942810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:58.978977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:58.988943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:58.988943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:58.988943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:58.988943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:58.988943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:58.995804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 914/1855] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:58 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:59.002617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:59.018836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a820dd8-42c8-4f59-a2f3-4c7dcb05671c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:59.025648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:59.028148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:59.028525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:59.030803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:59.030803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:59.032920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:59.055729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 942/1856] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:18:59 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:18:59.059780 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:59.070224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700c00a5-09a3-4cd4-81c4-9b825247d502 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:59.079080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3376c6-c384-41d4-a4cc-531bb55c09c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:59.080489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3376c6-c384-41d4-a4cc-531bb55c09c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:59.080776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3376c6-c384-41d4-a4cc-531bb55c09c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:59.081059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3376c6-c384-41d4-a4cc-531bb55c09c7 admin admin] PATH_INFO: `/v3/projects/63c23ac9cde849a6b42c5e52f2d3cdd7/users/aabee0ed08754c29ba98f21c5e2a0ce8/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:59.100232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3376c6-c384-41d4-a4cc-531bb55c09c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=63c23ac9cde849a6b42c5e52f2d3cdd7, user_id=aabee0ed08754c29ba98f21c5e2a0ce8, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:59.103251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3376c6-c384-41d4-a4cc-531bb55c09c7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:59.155583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:18:59.207704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 915/1857] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:18:59 2022] PUT /identity/v3/projects/63c23ac9cde849a6b42c5e52f2d3cdd7/users/aabee0ed08754c29ba98f21c5e2a0ce8/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:18:59.359403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:59.359403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:59.360084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa339f2-b424-4557-8e92-c9da2afe44fc None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:59.360084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa339f2-b424-4557-8e92-c9da2afe44fc None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:59.360084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aa339f2-b424-4557-8e92-c9da2afe44fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:59.361584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db7a41b-be5e-47ee-88a5-09a185d589fc admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:59.365792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3aa339f2-b424-4557-8e92-c9da2afe44fc None None] MFA Rules not processed for user `aabee0ed08754c29ba98f21c5e2a0ce8`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:18:59.365792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3aa339f2-b424-4557-8e92-c9da2afe44fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:18:59.365962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:59.365962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:59.365962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:59.365962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] PATH_INFO: `/v3/projects/7a4d418f00c74c7ea11f8eb2ba4898c1` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:59.366294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7a4d418f00c74c7ea11f8eb2ba4898c1)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:59.366294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:18:59.366481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 916/1858] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:18:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:18:59.395052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa339f2-b424-4557-8e92-c9da2afe44fc None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:18:59.408505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aa339f2-b424-4557-8e92-c9da2afe44fc None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:18:59.422602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:18:59.423348 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:18:59.423765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:18:59.423881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:18:59.424551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:18:59.426590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:00.165510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7a4d418f00c74c7ea11f8eb2ba4898c1', 'request_id': 'req-a1798a09-c4aa-460f-9036-edc69fd15209'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.170707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 917/1859] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:18:59 2022] POST /identity/v3/projects => generated 385 bytes in 789 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:00.175180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:00.256567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7a4d418f00c74c7ea11f8eb2ba4898c1', 'request_id': 'req-a1798a09-c4aa-460f-9036-edc69fd15209'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.269065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7a4d418f00c74c7ea11f8eb2ba4898c1', 'request_id': 'req-a1798a09-c4aa-460f-9036-edc69fd15209'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.282205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] The token cache is being invalidate because project 7a4d418f00c74c7ea11f8eb2ba4898c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:00.283175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.283659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.286307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 943/1860] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:18:59 2022] DELETE /identity/v3/projects/7a4d418f00c74c7ea11f8eb2ba4898c1 => generated 0 bytes in 1069 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:00.287769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:00.289834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:00.348417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1677f232-03b0-4e55-8e30-15baca4df764 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:00.360106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e05687d-e843-4608-891a-1aeb7209089c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:00.360800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05687d-e843-4608-891a-1aeb7209089c admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.361087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05687d-e843-4608-891a-1aeb7209089c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.361365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e05687d-e843-4608-891a-1aeb7209089c admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.362065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e05687d-e843-4608-891a-1aeb7209089c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:00.369296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e05687d-e843-4608-891a-1aeb7209089c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:00.392237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1798a09-c4aa-460f-9036-edc69fd15209 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:00.397371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:00.398538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.399019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.399298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] PATH_INFO: `/v3/users/37489203c7a94fe0b55cd1a57a72da92` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.400774 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] RBAC: Authorizing `identity:delete_user(user_id=37489203c7a94fe0b55cd1a57a72da92)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:00.402731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:00.525407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 918/1861] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:00 2022] POST /identity/v3/users => generated 293 bytes in 351 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:00.526739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.527053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.527337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.556260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:00.557501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:00.566368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 919/1862] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:00.567302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:00.576983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:00.589197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f5a8843-0a05-480e-8d72-483a60c8634f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:00.594938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab6cdab-604f-4f1c-9599-d62d589ba926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:00.595611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab6cdab-604f-4f1c-9599-d62d589ba926 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.595895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab6cdab-604f-4f1c-9599-d62d589ba926 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.596173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab6cdab-604f-4f1c-9599-d62d589ba926 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.596909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab6cdab-604f-4f1c-9599-d62d589ba926 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:00.599463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab6cdab-604f-4f1c-9599-d62d589ba926 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:00.615334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 920/1863] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:00 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:00.615334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03e1f0-1420-4459-9691-fa3d06ad03c1 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.615334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03e1f0-1420-4459-9691-fa3d06ad03c1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.615334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b03e1f0-1420-4459-9691-fa3d06ad03c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.643284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3b03e1f0-1420-4459-9691-fa3d06ad03c1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:00.644650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3b03e1f0-1420-4459-9691-fa3d06ad03c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:00.653791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 921/1864] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:00.654436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b03e1f0-1420-4459-9691-fa3d06ad03c1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:00.664506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b03e1f0-1420-4459-9691-fa3d06ad03c1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:00.671599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:00.677653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.677947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.678214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.679430 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:00.680716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:00.704853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 922/1865] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:00 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:00.705828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:00.719867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c24365f6-366e-4603-8217-650b8b3b0942 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:00.728551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:00.730264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.730264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.730264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.731119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:00.731399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:00.733640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:00.737100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '37489203c7a94fe0b55cd1a57a72da92'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.748753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 923/1866] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:00.749667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:00.760733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-486852b6-1cad-4469-bd2d-89bd964ecfd9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:00.768294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8b3a24-2fa2-47a8-9ab5-dbb51bf23d1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:00.769196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b3a24-2fa2-47a8-9ab5-dbb51bf23d1b admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.769484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b3a24-2fa2-47a8-9ab5-dbb51bf23d1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.769767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8b3a24-2fa2-47a8-9ab5-dbb51bf23d1b admin admin] PATH_INFO: `/v3/projects/6c9c767a1d974c84936583817c9fb6e6/users/f5717d8d55744a019a6861fda1d4743b/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:00.788415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b3a24-2fa2-47a8-9ab5-dbb51bf23d1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c9c767a1d974c84936583817c9fb6e6, user_id=f5717d8d55744a019a6861fda1d4743b, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:00.791017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8b3a24-2fa2-47a8-9ab5-dbb51bf23d1b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:00.868518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '37489203c7a94fe0b55cd1a57a72da92'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.877824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '37489203c7a94fe0b55cd1a57a72da92'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.884860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '37489203c7a94fe0b55cd1a57a72da92'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.884860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '37489203c7a94fe0b55cd1a57a72da92'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:00.902791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 944/1867] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:00 2022] DELETE /identity/v3/users/37489203c7a94fe0b55cd1a57a72da92 => generated 0 bytes in 616 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:00.905652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:00.905652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:00.962600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 924/1868] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:00 2022] PUT /identity/v3/projects/6c9c767a1d974c84936583817c9fb6e6/users/f5717d8d55744a019a6861fda1d4743b/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:00.962600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117dde89-5bec-4d16-b795-3efe77c57725 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:00.962600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117dde89-5bec-4d16-b795-3efe77c57725 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:00.962600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-117dde89-5bec-4d16-b795-3efe77c57725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.005932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-117dde89-5bec-4d16-b795-3efe77c57725 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:01.007436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-117dde89-5bec-4d16-b795-3efe77c57725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:01.021168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edbfdf98-4b13-49e4-913f-be78b9793e15 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.090918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 925/1869] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:01.092763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.093158 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.093663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.096311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57415930-d789-4ca4-a1f0-41a20094c2f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.100626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57415930-d789-4ca4-a1f0-41a20094c2f4 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.100626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57415930-d789-4ca4-a1f0-41a20094c2f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.100626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57415930-d789-4ca4-a1f0-41a20094c2f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.100626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57415930-d789-4ca4-a1f0-41a20094c2f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.101196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57415930-d789-4ca4-a1f0-41a20094c2f4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.124292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] MFA Rules not processed for user `f5717d8d55744a019a6861fda1d4743b`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:01.128691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:01.193124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 926/1870] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:01.193993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.197619 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:01.224815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 945/1871] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:00 2022] POST /identity/v3/projects => generated 373 bytes in 321 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:01.234280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39d4309-4d7b-4302-8f67-c5f7f993a429 None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.234280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39d4309-4d7b-4302-8f67-c5f7f993a429 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.234280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39d4309-4d7b-4302-8f67-c5f7f993a429 None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.234280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 946/1872] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:19:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:19:01.234280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39d4309-4d7b-4302-8f67-c5f7f993a429 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.235107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e39d4309-4d7b-4302-8f67-c5f7f993a429 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:01.266749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf431e94-11b6-4a8e-9c08-3a8adce2359b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.325398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.326380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.326852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.327789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.391693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39d4309-4d7b-4302-8f67-c5f7f993a429 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.397098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ce2688-f99c-4e72-aa96-ee27cc12a89b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.397770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce2688-f99c-4e72-aa96-ee27cc12a89b admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.398054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce2688-f99c-4e72-aa96-ee27cc12a89b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.398473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ce2688-f99c-4e72-aa96-ee27cc12a89b admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.399191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ce2688-f99c-4e72-aa96-ee27cc12a89b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.401318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ce2688-f99c-4e72-aa96-ee27cc12a89b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.416868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.419974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.429891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 927/1873] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 236 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:01.430690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.441144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35690e5-c72b-4daa-9033-52212f9034a4 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.467598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.468637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.468930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.469214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.469956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.472309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.523317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 947/1874] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:01 2022] POST /identity/v3/projects => generated 387 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:01.523317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacee997-0e6c-47f8-ab60-9e12689a0f55 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.523827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacee997-0e6c-47f8-ab60-9e12689a0f55 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.523827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacee997-0e6c-47f8-ab60-9e12689a0f55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.542026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bacee997-0e6c-47f8-ab60-9e12689a0f55 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:01.543922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bacee997-0e6c-47f8-ab60-9e12689a0f55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:01.581450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 928/1875] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:01 2022] POST /identity/v3/users => generated 287 bytes in 151 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:01.582520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.594982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c75fa78-b3cb-4dfa-a0cc-6b91be01a3ca admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.630849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 948/1876] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:19:01 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:01.631078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacee997-0e6c-47f8-ab60-9e12689a0f55 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.682806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.684251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.684551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.684833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.685541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.687639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.713883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacee997-0e6c-47f8-ab60-9e12689a0f55 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.720630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.721360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.721709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.728070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.728070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.728070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.741006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 949/1877] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:01 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:01.747168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.747168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:01.777440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 929/1878] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:01 2022] POST /identity/v3/users => generated 300 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:01.778724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.785591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24865bfa-8010-479b-a676-751d11eba05e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.795861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456cc642-7afd-41c9-a0b6-9278402e0cfa admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.802454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.803127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.803420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.803703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.813931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.814662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.815353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.815684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.816427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.818293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.829026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.829365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.838083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 930/1879] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:01 2022] GET /identity/v3/roles => generated 1420 bytes in 60 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:01.839463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.851320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a347e7a1-1597-4bd4-bee5-b8d0851471e1 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.856932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 950/1880] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:01 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:01.857903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.859098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.859858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.860220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.861212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.862034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.863784 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.870118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9ee2ea-1c85-4877-a3db-871cc906b9e7 service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.876819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50c718c4-6c22-46a7-89cf-bf279167d8e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.878308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c718c4-6c22-46a7-89cf-bf279167d8e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.878599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c718c4-6c22-46a7-89cf-bf279167d8e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.878929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50c718c4-6c22-46a7-89cf-bf279167d8e1 admin admin] PATH_INFO: `/v3/projects/8bf2b73c991f489d915d7124b5d24564/users/3ccee18a56e44ccebfa14d264deb219b/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.881700 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 931/1881] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:01 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:01.882837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:01.898809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392c91ca-a1bb-4c7b-8f9f-5cd5de2fd33a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:01.902231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c718c4-6c22-46a7-89cf-bf279167d8e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8bf2b73c991f489d915d7124b5d24564, user_id=3ccee18a56e44ccebfa14d264deb219b, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.906050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50c718c4-6c22-46a7-89cf-bf279167d8e1 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:01.907147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:01.908693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:01.908979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:01.909260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] PATH_INFO: `/v3/projects/c2e320fa8403467d9b348ebeb3061086/users/c2d67b8824a043fdb75569e078ed4fda/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:01.926209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2e320fa8403467d9b348ebeb3061086, user_id=c2d67b8824a043fdb75569e078ed4fda, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:01.929738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:02.016877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 951/1882] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:01 2022] PUT /identity/v3/projects/8bf2b73c991f489d915d7124b5d24564/users/3ccee18a56e44ccebfa14d264deb219b/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:02.022399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b916a2-f92c-4202-b6f3-b80ffa18a739 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.025966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b916a2-f92c-4202-b6f3-b80ffa18a739 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.025966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b916a2-f92c-4202-b6f3-b80ffa18a739 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.040379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a5b916a2-f92c-4202-b6f3-b80ffa18a739 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:02.041637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a5b916a2-f92c-4202-b6f3-b80ffa18a739 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:02.057263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:02.078954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 932/1883] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:01 2022] PUT /identity/v3/projects/c2e320fa8403467d9b348ebeb3061086/users/c2d67b8824a043fdb75569e078ed4fda/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:02.078954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:02.085949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:02.132859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 952/1884] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:02.133911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b916a2-f92c-4202-b6f3-b80ffa18a739 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:02.144801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b916a2-f92c-4202-b6f3-b80ffa18a739 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:02.176992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a337b389-f770-4dea-a314-8c39f3d79b6d admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:02.182631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:02.183534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.183855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.184133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] PATH_INFO: `/v3/projects/7efe566f45cd4e3ea84901d0439373c2` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.185528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7efe566f45cd4e3ea84901d0439373c2)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:02.187503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:02.212005 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:02.216476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:02.217188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.217475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.217756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.218918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:02.221391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:02.238315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 953/1885] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:02 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:02.238315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:02.238315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:02.295923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cdea3b-24ea-4525-993e-04b0853d7393 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:02.308137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7efe566f45cd4e3ea84901d0439373c2', 'request_id': 'req-64a0e200-1396-4479-81c6-319a84386008'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.313712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7efe566f45cd4e3ea84901d0439373c2', 'request_id': 'req-64a0e200-1396-4479-81c6-319a84386008'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.344652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:02.345333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.346020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.346337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.399323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7efe566f45cd4e3ea84901d0439373c2', 'request_id': 'req-64a0e200-1396-4479-81c6-319a84386008'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.409209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:02.410661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:02.411813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] The token cache is being invalidate because project 7efe566f45cd4e3ea84901d0439373c2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:02.412166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.412873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.415803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 933/1886] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:02 2022] DELETE /identity/v3/projects/7efe566f45cd4e3ea84901d0439373c2 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:02.421793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:02.421793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:02.433606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a0e200-1396-4479-81c6-319a84386008 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:02.435380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 954/1887] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:02.436469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:02.439135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ac8fea-db63-4e13-b609-65140b04690b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:02.440102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac8fea-db63-4e13-b609-65140b04690b admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.440642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac8fea-db63-4e13-b609-65140b04690b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.440901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ac8fea-db63-4e13-b609-65140b04690b admin admin] PATH_INFO: `/v3/projects/c2e320fa8403467d9b348ebeb3061086/users/c2d67b8824a043fdb75569e078ed4fda/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.443070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ac8fea-db63-4e13-b609-65140b04690b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c2e320fa8403467d9b348ebeb3061086, user_id=c2d67b8824a043fdb75569e078ed4fda, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:02.445957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ac8fea-db63-4e13-b609-65140b04690b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:02.446968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b19f8583-dc63-4a56-9045-87ebd20d8fde service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:02.452672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:02.453570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.453848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.454124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] PATH_INFO: `/v3/users/d8c167cd14d44ab9b9883e9b1c6f24d2` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.456003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d8c167cd14d44ab9b9883e9b1c6f24d2)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:02.457929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:02.458328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 934/1888] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:02 2022] PUT /identity/v3/projects/c2e320fa8403467d9b348ebeb3061086/users/c2d67b8824a043fdb75569e078ed4fda/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:02.471490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.471785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.472077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.499604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] MFA Rules not processed for user `c2d67b8824a043fdb75569e078ed4fda`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:02.500833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:02.573522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 935/1889] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:02 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:02.582670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:02.584443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:02.643569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d99a458-5edd-4aaf-9d0b-99d6f229a3e6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:02.647234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:02.647885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:02.648170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:02.648450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:02.660211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:02.660968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:02.667834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd8c167cd14d44ab9b9883e9b1c6f24d2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.683246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 936/1890] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:02 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:19:02.759932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd8c167cd14d44ab9b9883e9b1c6f24d2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.773206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd8c167cd14d44ab9b9883e9b1c6f24d2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.773955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd8c167cd14d44ab9b9883e9b1c6f24d2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.774566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ec38af5a-5b87-42e2-83db-f588bac26a44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd8c167cd14d44ab9b9883e9b1c6f24d2'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:02.794313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 955/1891] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:02 2022] DELETE /identity/v3/users/d8c167cd14d44ab9b9883e9b1c6f24d2 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:03.730794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:03.732872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:03.806651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bbd303-a388-4bb6-98bd-9587b0208e97 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:03.872314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:03.874888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:03.874888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:03.874888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] PATH_INFO: `/v3/projects/e0baae198b7749d199d63a41f25f828e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:03.875169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e0baae198b7749d199d63a41f25f828e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:03.876896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:03.926272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:04.030272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e0baae198b7749d199d63a41f25f828e', 'request_id': 'req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:04.036589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e0baae198b7749d199d63a41f25f828e', 'request_id': 'req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:04.119514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e0baae198b7749d199d63a41f25f828e', 'request_id': 'req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:04.137600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] The token cache is being invalidate because project e0baae198b7749d199d63a41f25f828e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:04.138152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:04.139552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b5bfd4c2-e7fa-4c24-8dd7-407d288b5ac2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:04.141687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 937/1892] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:03 2022] DELETE /identity/v3/projects/e0baae198b7749d199d63a41f25f828e => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:04.146622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.147191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.147191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.165777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:04.167030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:04.262655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 956/1893] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:04.262759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9819b2a-0837-4c2d-8aa8-fd64f1377f2f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.262759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9819b2a-0837-4c2d-8aa8-fd64f1377f2f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.262759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9819b2a-0837-4c2d-8aa8-fd64f1377f2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.279765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.282406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:04.297029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d9819b2a-0837-4c2d-8aa8-fd64f1377f2f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:04.297029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d9819b2a-0837-4c2d-8aa8-fd64f1377f2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:04.305109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 938/1894] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:04.309764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9819b2a-0837-4c2d-8aa8-fd64f1377f2f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.319402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9819b2a-0837-4c2d-8aa8-fd64f1377f2f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.328620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d1957-d8e3-4dc9-8d31-86a31c959963 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.329308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d1957-d8e3-4dc9-8d31-86a31c959963 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.329599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d1957-d8e3-4dc9-8d31-86a31c959963 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.330360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d1957-d8e3-4dc9-8d31-86a31c959963 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.330605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d1957-d8e3-4dc9-8d31-86a31c959963 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:04.330943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d1957-d8e3-4dc9-8d31-86a31c959963 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.332701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d1957-d8e3-4dc9-8d31-86a31c959963 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:04.355296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 939/1895] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:19:04.357380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.357668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.357944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.365067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54a76aab-06ed-4446-8042-37cd7bab37fd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.369303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.369961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.370243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.370519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.373125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:04.392558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] MFA Rules not processed for user `3ccee18a56e44ccebfa14d264deb219b`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:04.397069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:04.451413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.452948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:04.463747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 957/1896] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:04 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:04.464658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.476570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c9805af-4686-4c20-b8a3-ae6bebda12bf service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.480451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=82,keystone:DELETE=13,keystone:INSERT=5 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:04.488995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.489317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 940/1897] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:04 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:04.489414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.489414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.489414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.489414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.489414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:04.536377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.538059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:04.552303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b9395c-b844-4fec-aec8-e607fe45f009 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.568286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.568286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.568286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.568286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.568690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.568829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:04.579043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 941/1898] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:04 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:04.598750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 958/1899] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:04 2022] POST /identity/v3/projects => generated 407 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:04.602753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.618495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dac74aa8-3c54-4902-9f49-b12650506930 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.694831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.695520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.695814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.696097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.696812 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.698794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:04.792630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 942/1900] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:04 2022] POST /identity/v3/users => generated 304 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:04.794402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.806139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072084cf-7e48-4d64-b814-d8c5745f4d7f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.811221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.811894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.812179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.812473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.813206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.814865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:04.827165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 959/1901] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:04 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:04.831372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.842438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026494d6-4cd5-4c97-aad5-759015a2fca2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.847151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3124df3-391c-4b5d-b97d-9ddc74f3f109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.847879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3124df3-391c-4b5d-b97d-9ddc74f3f109 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.848198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3124df3-391c-4b5d-b97d-9ddc74f3f109 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.848531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3124df3-391c-4b5d-b97d-9ddc74f3f109 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.849333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3124df3-391c-4b5d-b97d-9ddc74f3f109 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.851122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3124df3-391c-4b5d-b97d-9ddc74f3f109 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:04.862020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 943/1902] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:04.865867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:04.876110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a4cafdb-30d6-44d7-a421-700696aba707 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:04.880758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:04.885397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:04.885397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:04.885397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] PATH_INFO: `/v3/projects/ddd9706b7b4f4852bcb6c5e2acbc8326/users/8515192d35174302afbc18400e80a232/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:04.898071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ddd9706b7b4f4852bcb6c5e2acbc8326, user_id=8515192d35174302afbc18400e80a232, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:04.900767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:05.000572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 960/1903] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:04 2022] PUT /identity/v3/projects/ddd9706b7b4f4852bcb6c5e2acbc8326/users/8515192d35174302afbc18400e80a232/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:05.020229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a80538-f971-4984-9d8c-3dc3698f9c83 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.020521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a80538-f971-4984-9d8c-3dc3698f9c83 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.020804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70a80538-f971-4984-9d8c-3dc3698f9c83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.051269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-70a80538-f971-4984-9d8c-3dc3698f9c83 None None] MFA Rules not processed for user `8515192d35174302afbc18400e80a232`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:05.056010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-70a80538-f971-4984-9d8c-3dc3698f9c83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:05.118451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 944/1904] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:05 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:05.123392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:05.193058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7845596e-3b4c-4772-ae8f-4a38b92b8638 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:05.197755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:05.198418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.198694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.198997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.200422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:05.202318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:05.329433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 961/1905] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:05 2022] POST /identity/v3/projects => generated 409 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:05.331148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a80538-f971-4984-9d8c-3dc3698f9c83 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:05.360153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70a80538-f971-4984-9d8c-3dc3698f9c83 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:05.377298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:05.438279 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:05.438996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.439312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.439603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.440264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:05.442295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:05.521334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 945/1906] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:05 2022] POST /identity/v3/users => generated 311 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:05.527118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:05.538727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4028a3b-93bc-40a9-af70-1073c0f742b9 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:05.547037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:05.547873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.548344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.548840 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.549823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:05.551905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:05.563903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 962/1907] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:05 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:05.568392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:05.580441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63ba6d5-25db-4633-b44b-f0278ef84747 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:05.585342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:05.586220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.586680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.587007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] PATH_INFO: `/v3/projects/a8133c23656941b2a73c0e5086623840/users/b9beb098afb8474eaa5ec653f7e50e2b/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.603854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8133c23656941b2a73c0e5086623840, user_id=b9beb098afb8474eaa5ec653f7e50e2b, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:05.606927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:05.679715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 946/1908] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:05 2022] PUT /identity/v3/projects/a8133c23656941b2a73c0e5086623840/users/b9beb098afb8474eaa5ec653f7e50e2b/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:05.683865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:05.694003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89257d81-c60e-4279-9687-c6933950b43c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:05.753805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6864ef30-f780-4892-8e32-e3e29ee92d56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:05.754507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864ef30-f780-4892-8e32-e3e29ee92d56 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.754820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864ef30-f780-4892-8e32-e3e29ee92d56 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.755104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6864ef30-f780-4892-8e32-e3e29ee92d56 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.755835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6864ef30-f780-4892-8e32-e3e29ee92d56 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:05.757510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6864ef30-f780-4892-8e32-e3e29ee92d56 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:05.769944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 963/1909] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:05 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:05.773928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:05.785507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86eeb47b-562f-4c23-9ae2-082230e4c3b2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:05.790239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:05.791185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.791469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.791870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] PATH_INFO: `/v3/projects/a8133c23656941b2a73c0e5086623840/users/b9beb098afb8474eaa5ec653f7e50e2b/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.795042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8133c23656941b2a73c0e5086623840, user_id=b9beb098afb8474eaa5ec653f7e50e2b, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:05.797547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:05.809716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 947/1910] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:05 2022] PUT /identity/v3/projects/a8133c23656941b2a73c0e5086623840/users/b9beb098afb8474eaa5ec653f7e50e2b/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:05.820186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e8b50d-af26-461a-911f-66215953b0aa None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.820477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e8b50d-af26-461a-911f-66215953b0aa None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.820801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61e8b50d-af26-461a-911f-66215953b0aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:05.849997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-61e8b50d-af26-461a-911f-66215953b0aa None None] MFA Rules not processed for user `b9beb098afb8474eaa5ec653f7e50e2b`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:05.851231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-61e8b50d-af26-461a-911f-66215953b0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:05.921504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 964/1911] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:05 2022] POST /identity/v3/auth/tokens => generated 3343 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:05.932096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:05.986758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313c8306-551b-48be-871a-97b18c293865 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:05.990548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95876012-3b6c-4420-9619-69f756e151ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:05.991236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95876012-3b6c-4420-9619-69f756e151ec service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:05.991514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95876012-3b6c-4420-9619-69f756e151ec service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:05.991877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95876012-3b6c-4420-9619-69f756e151ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:06.001463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95876012-3b6c-4420-9619-69f756e151ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:06.002930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95876012-3b6c-4420-9619-69f756e151ec service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:06.013202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 948/1912] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:05 2022] GET /identity/v3/auth/tokens => generated 3343 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:06.045749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:06.205547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:06.427180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e8b50d-af26-461a-911f-66215953b0aa None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:06.436714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61e8b50d-af26-461a-911f-66215953b0aa None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:06.440222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:06.440862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:06.441140 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:06.441413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:06.497817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:06.499342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:06.510092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 965/1913] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:06 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:06.610543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:06.696416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:06.748454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:08.033807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:08.147934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:08.889339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04531f9-f482-4856-a8c9-658782de38ed None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:08.889716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04531f9-f482-4856-a8c9-658782de38ed None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:08.890217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04531f9-f482-4856-a8c9-658782de38ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:08.909549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b04531f9-f482-4856-a8c9-658782de38ed None None] MFA Rules not processed for user `3ccee18a56e44ccebfa14d264deb219b`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:08.910832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b04531f9-f482-4856-a8c9-658782de38ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:08.982852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 949/1914] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:08 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:09.719986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:09.901941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:10.536814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:10.645494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:10.954689 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:10.988136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.102325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.251260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=93,keystone:DELETE=5,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.290651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.399313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.422225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.473517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.507837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.524984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:11.673575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:13.036185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:13.045224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aee0cf5-6073-4bd6-a19c-77f97b059d97 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:13.048741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:13.049389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:13.053777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:13.053777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:13.053777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:13.077667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:13.115292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:13.117084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:13.136434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 966/1915] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:13 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:19:18.405107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04531f9-f482-4856-a8c9-658782de38ed None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:18.426058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:18.464064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:18.471540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:18.507226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04531f9-f482-4856-a8c9-658782de38ed None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:18.529063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:18.529063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:18.529063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:18.529063 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] PATH_INFO: `/v3/users/8515192d35174302afbc18400e80a232` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:18.529806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8515192d35174302afbc18400e80a232)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:18.531828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:18.612711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f861f2dd-42e7-4ee5-884e-bf5f4547f26c service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:18.630860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:18.633987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:18.634401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:18.635464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] PATH_INFO: `/v3/users/aabee0ed08754c29ba98f21c5e2a0ce8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:18.639262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=aabee0ed08754c29ba98f21c5e2a0ce8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:18.642360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:18.889062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'aabee0ed08754c29ba98f21c5e2a0ce8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:18.898834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8515192d35174302afbc18400e80a232'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:18.976442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'aabee0ed08754c29ba98f21c5e2a0ce8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:18.986306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'aabee0ed08754c29ba98f21c5e2a0ce8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:18.987759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'aabee0ed08754c29ba98f21c5e2a0ce8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:18.988815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f0e17615-ebf0-4c47-aab1-bb46be8b75a9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'aabee0ed08754c29ba98f21c5e2a0ce8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:19.033280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8515192d35174302afbc18400e80a232'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:19.034254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8515192d35174302afbc18400e80a232'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:19.036597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 967/1916] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:18 2022] DELETE /identity/v3/users/aabee0ed08754c29ba98f21c5e2a0ce8 => generated 0 bytes in 574 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:19.041071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb686207-c9f5-4969-beac-37026b5527df None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:19.042158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb686207-c9f5-4969-beac-37026b5527df None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:19.042158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb686207-c9f5-4969-beac-37026b5527df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:19.049353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8515192d35174302afbc18400e80a232'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:19.050339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21e48e39-5068-4271-8392-92acf1558126 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8515192d35174302afbc18400e80a232'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:19.089737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 950/1917] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:18 2022] DELETE /identity/v3/users/8515192d35174302afbc18400e80a232 => generated 0 bytes in 685 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:19.091441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bb686207-c9f5-4969-beac-37026b5527df None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:19.093950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050c6c1-39bf-4846-8ed4-a79e07c2a9fd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:19.094114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bb686207-c9f5-4969-beac-37026b5527df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:19.095166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050c6c1-39bf-4846-8ed4-a79e07c2a9fd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:19.095740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050c6c1-39bf-4846-8ed4-a79e07c2a9fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:19.129016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3050c6c1-39bf-4846-8ed4-a79e07c2a9fd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:19.131676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3050c6c1-39bf-4846-8ed4-a79e07c2a9fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:19.206487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:19.268536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 968/1918] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:19.291861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 951/1919] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:19.299445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb686207-c9f5-4969-beac-37026b5527df None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:19.300977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bb686207-c9f5-4969-beac-37026b5527df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:19.395832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb686207-c9f5-4969-beac-37026b5527df None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:19.400206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:19.401984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:19.401984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:19.401984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:19.415560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050c6c1-39bf-4846-8ed4-a79e07c2a9fd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:19.438524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050c6c1-39bf-4846-8ed4-a79e07c2a9fd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:19.440991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:19.442733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:19.444646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:19.448076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:19.448501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:19.450498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:19.459733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 969/1920] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:19.472845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:19.475701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:19.493856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 952/1921] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:20.424000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:20.425598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:20.426234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:20.428306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:20.448337 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b01f98a-2035-4014-9d76-175efeb73f82 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:20.450800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902cce43-7c9b-4e59-817a-e7792c7695a7 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:20.460488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:20.461466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:20.464433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:20.465135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:20.465328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] PATH_INFO: `/v3/projects/ddd9706b7b4f4852bcb6c5e2acbc8326` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:20.465879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:20.466487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:20.467007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] PATH_INFO: `/v3/projects/63c23ac9cde849a6b42c5e52f2d3cdd7` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:20.469236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] RBAC: Authorizing `identity:delete_project(project_id=63c23ac9cde849a6b42c5e52f2d3cdd7)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:20.470176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] RBAC: Authorizing `identity:delete_project(project_id=ddd9706b7b4f4852bcb6c5e2acbc8326)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:20.472132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:20.473202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:20.507850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:20.599451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '63c23ac9cde849a6b42c5e52f2d3cdd7', 'request_id': 'req-4bc3b212-e7e3-4214-abdb-c43f97767677'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.609818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '63c23ac9cde849a6b42c5e52f2d3cdd7', 'request_id': 'req-4bc3b212-e7e3-4214-abdb-c43f97767677'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.681477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '63c23ac9cde849a6b42c5e52f2d3cdd7', 'request_id': 'req-4bc3b212-e7e3-4214-abdb-c43f97767677'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.682924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ddd9706b7b4f4852bcb6c5e2acbc8326', 'request_id': 'req-7f619ebf-1eb6-4c57-8273-4163889541bd'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.697419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] The token cache is being invalidate because project 63c23ac9cde849a6b42c5e52f2d3cdd7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:20.697957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.698619 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.701205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 953/1922] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:20 2022] DELETE /identity/v3/projects/63c23ac9cde849a6b42c5e52f2d3cdd7 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:20.705010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:20.706500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:20.784646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc3b212-e7e3-4214-abdb-c43f97767677 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:20.799950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ddd9706b7b4f4852bcb6c5e2acbc8326', 'request_id': 'req-7f619ebf-1eb6-4c57-8273-4163889541bd'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.810601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ddd9706b7b4f4852bcb6c5e2acbc8326', 'request_id': 'req-7f619ebf-1eb6-4c57-8273-4163889541bd'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.822979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] The token cache is being invalidate because project ddd9706b7b4f4852bcb6c5e2acbc8326 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:20.823584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.825103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:20.827142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 970/1923] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:20 2022] DELETE /identity/v3/projects/ddd9706b7b4f4852bcb6c5e2acbc8326 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:20.831000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:20.832900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:20.857540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:20.858336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:20.858694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:20.858926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] PATH_INFO: `/v3/users/f5717d8d55744a019a6861fda1d4743b` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:20.860695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5717d8d55744a019a6861fda1d4743b)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:20.862649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:20.915174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f619ebf-1eb6-4c57-8273-4163889541bd admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:20.920612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:20.921624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:20.922027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:20.922375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] PATH_INFO: `/v3/users/b9beb098afb8474eaa5ec653f7e50e2b` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:20.923866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b9beb098afb8474eaa5ec653f7e50e2b)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:20.925821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:20.939448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:21.094807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5717d8d55744a019a6861fda1d4743b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.173169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b9beb098afb8474eaa5ec653f7e50e2b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.176300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5717d8d55744a019a6861fda1d4743b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.177292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5717d8d55744a019a6861fda1d4743b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.193388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5717d8d55744a019a6861fda1d4743b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.200676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5717d8d55744a019a6861fda1d4743b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.226226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 954/1924] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:20 2022] DELETE /identity/v3/users/f5717d8d55744a019a6861fda1d4743b => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:21.302479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b9beb098afb8474eaa5ec653f7e50e2b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.327212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b9beb098afb8474eaa5ec653f7e50e2b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.328030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b9beb098afb8474eaa5ec653f7e50e2b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.330055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b9beb098afb8474eaa5ec653f7e50e2b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:21.369752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 971/1925] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:20 2022] DELETE /identity/v3/users/b9beb098afb8474eaa5ec653f7e50e2b => generated 0 bytes in 539 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:21.737021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:21.779433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:21.779433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:21.858487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-433b4f53-80c8-443e-96ea-a1f06d9e6418 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:21.920787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:21.921720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:21.922012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:21.922286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] PATH_INFO: `/v3/projects/6c9c767a1d974c84936583817c9fb6e6` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:21.923534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c9c767a1d974c84936583817c9fb6e6)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:21.925475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:22.068230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c9c767a1d974c84936583817c9fb6e6', 'request_id': 'req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.074053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c9c767a1d974c84936583817c9fb6e6', 'request_id': 'req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.153635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c9c767a1d974c84936583817c9fb6e6', 'request_id': 'req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.165836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] The token cache is being invalidate because project 6c9c767a1d974c84936583817c9fb6e6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:22.166120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.166937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.172463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 955/1926] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:21 2022] DELETE /identity/v3/projects/6c9c767a1d974c84936583817c9fb6e6 => generated 0 bytes in 397 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:22.173745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:22.175600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:22.199199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68c6fcd-647b-49b3-8c8e-a4bb4ed65e11 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:22.247505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:22.252038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:22.265720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:22.266729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:22.267109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:22.267457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] PATH_INFO: `/v3/users/c2d67b8824a043fdb75569e078ed4fda` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:22.269422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2d67b8824a043fdb75569e078ed4fda)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:22.271961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:22.346589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e08bde7b-b5cb-4934-9bdd-a5bf2b9a6a5d admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:22.353184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:22.354137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:22.354422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:22.354697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] PATH_INFO: `/v3/projects/a8133c23656941b2a73c0e5086623840` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:22.356788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8133c23656941b2a73c0e5086623840)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:22.358879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:22.518061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8133c23656941b2a73c0e5086623840', 'request_id': 'req-46d0de51-829a-474f-84c9-0d50c11bc15b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.518307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2d67b8824a043fdb75569e078ed4fda'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.525129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8133c23656941b2a73c0e5086623840', 'request_id': 'req-46d0de51-829a-474f-84c9-0d50c11bc15b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.565112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2d67b8824a043fdb75569e078ed4fda'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.574272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2d67b8824a043fdb75569e078ed4fda'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.575001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2d67b8824a043fdb75569e078ed4fda'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.575666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9234f29d-c28b-4fb1-bdee-7a0fb1b137ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2d67b8824a043fdb75569e078ed4fda'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.596484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 972/1927] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:22 2022] DELETE /identity/v3/users/c2d67b8824a043fdb75569e078ed4fda => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:22.627942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8133c23656941b2a73c0e5086623840', 'request_id': 'req-46d0de51-829a-474f-84c9-0d50c11bc15b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.641962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] The token cache is being invalidate because project a8133c23656941b2a73c0e5086623840 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:22.642260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.643118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-46d0de51-829a-474f-84c9-0d50c11bc15b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:22.644716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 956/1928] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:22 2022] DELETE /identity/v3/projects/a8133c23656941b2a73c0e5086623840 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:22.649766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146024ab-af2f-4244-b287-ec207b20201b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:22.650150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146024ab-af2f-4244-b287-ec207b20201b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:22.650449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146024ab-af2f-4244-b287-ec207b20201b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:22.670385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-146024ab-af2f-4244-b287-ec207b20201b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:22.671632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-146024ab-af2f-4244-b287-ec207b20201b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:22.736656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:22.768221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 973/1929] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:22.776122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c9abde-3dbf-4469-b056-b28f37f8714c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:22.776570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c9abde-3dbf-4469-b056-b28f37f8714c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:22.776694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c9abde-3dbf-4469-b056-b28f37f8714c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:22.801535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b8c9abde-3dbf-4469-b056-b28f37f8714c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:22.804875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b8c9abde-3dbf-4469-b056-b28f37f8714c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:22.817219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 957/1930] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:22.818180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146024ab-af2f-4244-b287-ec207b20201b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:22.828725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146024ab-af2f-4244-b287-ec207b20201b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:22.833712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:22.834385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:22.834732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:22.835078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:22.835931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:22.836249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:22.837906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:22.850601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 974/1931] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:19:22.856505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c9abde-3dbf-4469-b056-b28f37f8714c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:22.867012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c9abde-3dbf-4469-b056-b28f37f8714c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:22.880577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:22.880577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:22.880577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:22.880577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:22.880981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:22.880981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.036899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 958/1932] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:22 2022] POST /identity/v3/projects => generated 383 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:23.041191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.051769 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11c7a449-17a2-41ba-a658-fff186ca69f0 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.107482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.108173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.108455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.108735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.109424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.111364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.132151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.133854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:23.149718 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fbb569a-9149-424d-934f-42a07c2b1c6d admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.155197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.156037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.156318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.156590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] PATH_INFO: `/v3/projects/c2e320fa8403467d9b348ebeb3061086` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.157818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c2e320fa8403467d9b348ebeb3061086)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.159848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.188655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 975/1933] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/users => generated 292 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:23.192630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.203684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7f0ad2-801c-40b6-b5bf-bfe351c8ce0b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.208374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.209034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.209364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.209632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.210382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.212044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.229372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 976/1934] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:23 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:23.229372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.243764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031a0fd1-5094-4dc4-aaff-f38c43820382 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.259905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c2e320fa8403467d9b348ebeb3061086', 'request_id': 'req-9fd7d973-3b76-4fa8-b84c-73acd49e174c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:23.268725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c2e320fa8403467d9b348ebeb3061086', 'request_id': 'req-9fd7d973-3b76-4fa8-b84c-73acd49e174c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:23.291194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:23.312698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.313354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.313638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.313914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.314646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.316326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.326713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 977/1935] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:23 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:23.331315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.332132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c2e320fa8403467d9b348ebeb3061086', 'request_id': 'req-9fd7d973-3b76-4fa8-b84c-73acd49e174c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:23.342352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd89115f-c81d-4162-89fa-e8188b9c6499 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.345628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] The token cache is being invalidate because project c2e320fa8403467d9b348ebeb3061086 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:23.345923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:23.346677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9fd7d973-3b76-4fa8-b84c-73acd49e174c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:23.347496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff9b327-f1f4-47c6-8a3f-7091cd5591ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.348278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 959/1936] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:23 2022] DELETE /identity/v3/projects/c2e320fa8403467d9b348ebeb3061086 => generated 0 bytes in 217 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:23.348680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9b327-f1f4-47c6-8a3f-7091cd5591ca admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.349038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9b327-f1f4-47c6-8a3f-7091cd5591ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.349250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff9b327-f1f4-47c6-8a3f-7091cd5591ca admin admin] PATH_INFO: `/v3/projects/785745b770024585902d82bdffdff518/users/e41838459e404a8eb591ba43f82c8985/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.354371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2560a-73f4-4ff6-bd70-048c7b95e374 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.354658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2560a-73f4-4ff6-bd70-048c7b95e374 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.354975 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec2560a-73f4-4ff6-bd70-048c7b95e374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.371581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff9b327-f1f4-47c6-8a3f-7091cd5591ca admin admin] RBAC: Authorizing `identity:create_grant(project_id=785745b770024585902d82bdffdff518, user_id=e41838459e404a8eb591ba43f82c8985, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.374207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-bec2560a-73f4-4ff6-bd70-048c7b95e374 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:23.374679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff9b327-f1f4-47c6-8a3f-7091cd5591ca admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.375455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-bec2560a-73f4-4ff6-bd70-048c7b95e374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:23.383608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 960/1937] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:23.391987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc59367-bf15-45be-a513-95412ffe0956 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.392278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc59367-bf15-45be-a513-95412ffe0956 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.392556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc59367-bf15-45be-a513-95412ffe0956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.411286 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4fc59367-bf15-45be-a513-95412ffe0956 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:23.412308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4fc59367-bf15-45be-a513-95412ffe0956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:23.421385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 961/1938] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:23.425346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc59367-bf15-45be-a513-95412ffe0956 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.434437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc59367-bf15-45be-a513-95412ffe0956 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.448479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 978/1939] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:23 2022] PUT /identity/v3/projects/785745b770024585902d82bdffdff518/users/e41838459e404a8eb591ba43f82c8985/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:23.454090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.454375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.454655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.495792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] MFA Rules not processed for user `e41838459e404a8eb591ba43f82c8985`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:23.497566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:23.499606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.503111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.503111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.503111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.503111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:23.503111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.504176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.516634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 962/1940] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:19:23.521136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.534716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01279379-0ab5-4507-9b52-21c0ce232694 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.544735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.545395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.545681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.545962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.547637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.550205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.566914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 979/1941] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:23.571523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.572867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:23.586618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-944d0ad5-fe81-4f6b-ac99-d67b7c71a593 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.591818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.592523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.592807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.593084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.593791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.595723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.622079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:23.624028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 963/1942] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/projects => generated 393 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:23.628093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.678288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 980/1943] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/projects => generated 381 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:23.679538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.689747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a85dde-fc11-4d93-b52f-4baadd46ced8 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.747998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd37584-ca9f-49ff-b863-ac3e955ec4be admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.754150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.754922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.755666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.755666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.756380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.758431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.765142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.765818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.766104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.766384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.767136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.769084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.847073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 981/1944] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/users => generated 297 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:23.850718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.855665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:23.861357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214d8ad-72fc-4733-b65b-2637b718f2c7 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.875711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 964/1945] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:23 2022] POST /identity/v3/users => generated 297 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:23.876059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.876059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.876059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.876059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.876382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.876382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.878567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.890506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd92778c-d069-4cb6-b0e3-51381625ec53 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.895168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 982/1946] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:23 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:23.895443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.896137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.896421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.896696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.897588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.899253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.905117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.917187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-356fad79-1fb9-4a92-94e5-5c2437360ee7 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.918206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 965/1947] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:23 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:23.922609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.926500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.928233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.928527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.928963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] PATH_INFO: `/v3/projects/ae363ceb53364d71bbf0831279347fbf/users/5b18aec5f7a64c2399cb97e521b7c51b/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.934029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-460cb36c-3e90-4d98-b0d3-3f6962f27353 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.943909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.944938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.945365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.945857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:23.946913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.948682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.961273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae363ceb53364d71bbf0831279347fbf, user_id=5b18aec5f7a64c2399cb97e521b7c51b, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:23.967071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:23.970376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 966/1948] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:23 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:23.979750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:23.989883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30fe454c-378e-4174-8618-9e47c1365f60 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:23.994417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c76373-0310-46c3-8941-ee8c6c6322c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:23.995331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c76373-0310-46c3-8941-ee8c6c6322c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:23.995615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c76373-0310-46c3-8941-ee8c6c6322c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:23.996399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c76373-0310-46c3-8941-ee8c6c6322c2 admin admin] PATH_INFO: `/v3/projects/286fd5b29ea94d898673abfb9d029fa7/users/2212c1f9c30045efabb132144cb1f1e0/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.010421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c76373-0310-46c3-8941-ee8c6c6322c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=286fd5b29ea94d898673abfb9d029fa7, user_id=2212c1f9c30045efabb132144cb1f1e0, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.013828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96c76373-0310-46c3-8941-ee8c6c6322c2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.056681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 983/1949] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:23 2022] PUT /identity/v3/projects/ae363ceb53364d71bbf0831279347fbf/users/5b18aec5f7a64c2399cb97e521b7c51b/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:24.060497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.079607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 967/1950] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:23 2022] PUT /identity/v3/projects/286fd5b29ea94d898673abfb9d029fa7/users/2212c1f9c30045efabb132144cb1f1e0/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:24.094584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa75dc4-c32a-4176-886d-29ebbc47518e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.095038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa75dc4-c32a-4176-886d-29ebbc47518e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.095335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa75dc4-c32a-4176-886d-29ebbc47518e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.125913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:24.130151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93136d72-58aa-4d93-9382-b3e2ceef2acc admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.132619 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-aaa75dc4-c32a-4176-886d-29ebbc47518e None None] MFA Rules not processed for user `2212c1f9c30045efabb132144cb1f1e0`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:24.133924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-aaa75dc4-c32a-4176-886d-29ebbc47518e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:24.187352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.188185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.188612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.188855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.189585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.191210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.200702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 968/1951] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:24 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:24.204474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 984/1952] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:24 2022] GET /identity/v3/roles => generated 1420 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:24.205428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.205567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa75dc4-c32a-4176-886d-29ebbc47518e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.215744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b635a23-8072-4449-bb68-0dce58b3df9b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.216409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa75dc4-c32a-4176-886d-29ebbc47518e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.220295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.220941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.222755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4c2576-c223-4563-9cae-533245995804 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.222899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.222899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.223060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4c2576-c223-4563-9cae-533245995804 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.223156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.223237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4c2576-c223-4563-9cae-533245995804 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.223237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4c2576-c223-4563-9cae-533245995804 admin admin] PATH_INFO: `/v3/projects/ae363ceb53364d71bbf0831279347fbf/users/5b18aec5f7a64c2399cb97e521b7c51b/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.224389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.224678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4c2576-c223-4563-9cae-533245995804 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae363ceb53364d71bbf0831279347fbf, user_id=5b18aec5f7a64c2399cb97e521b7c51b, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.227454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4c2576-c223-4563-9cae-533245995804 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.238685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 969/1953] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:24 2022] PUT /identity/v3/projects/ae363ceb53364d71bbf0831279347fbf/users/5b18aec5f7a64c2399cb97e521b7c51b/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:24.255492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5749814e-a98f-4a1d-8b2f-d41799d28056 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.255779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5749814e-a98f-4a1d-8b2f-d41799d28056 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.256078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5749814e-a98f-4a1d-8b2f-d41799d28056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.283905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-5749814e-a98f-4a1d-8b2f-d41799d28056 None None] MFA Rules not processed for user `5b18aec5f7a64c2399cb97e521b7c51b`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:24.285083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5749814e-a98f-4a1d-8b2f-d41799d28056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:24.353915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 970/1954] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:24 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:24.361359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 985/1955] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:24 2022] POST /identity/v3/projects => generated 391 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:24.365696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5749814e-a98f-4a1d-8b2f-d41799d28056 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.372449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.375199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:24.435694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5749814e-a98f-4a1d-8b2f-d41799d28056 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.437116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cdabd6e-d868-4992-9d30-fcdd4c703bbf admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.441652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.442306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.442582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.442900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.498173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.498887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.499312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.499594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.500303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.502247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.505022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:24.525789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:24.529076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.530650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.549597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 986/1956] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:24 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:19:24.580490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 971/1957] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:24 2022] POST /identity/v3/users => generated 302 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:24.581928 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.593595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d3b780-ab4c-4d38-b4b0-1e0ce804cc29 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.598359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.599060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.599342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.599616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.600347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.601992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.612700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 987/1958] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:24.616901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.626875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b99d0f-8e9d-470c-9c8a-34845a0399dc admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.631434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.632335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.632613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.632887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] PATH_INFO: `/v3/projects/aa8ecbdfe94c442cbaf3daf8b422609b/users/9c3872da38ab434aba83fbfeb440e085/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.647985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa8ecbdfe94c442cbaf3daf8b422609b, user_id=9c3872da38ab434aba83fbfeb440e085, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.650943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.681455 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.692146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee38f581-49d5-4773-af2f-f1aa97510562 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.697930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.697930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.697930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.697930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.754355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 972/1959] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:24 2022] PUT /identity/v3/projects/aa8ecbdfe94c442cbaf3daf8b422609b/users/9c3872da38ab434aba83fbfeb440e085/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:24.758234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.765084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.766534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.775712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 988/1960] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:24 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:19:24.822057 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05480ccf-4076-497c-8669-1f8e7499f49f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.827538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b58e604-89bf-4069-b513-3f6e776d2b15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.828205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b58e604-89bf-4069-b513-3f6e776d2b15 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.828487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b58e604-89bf-4069-b513-3f6e776d2b15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.828762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b58e604-89bf-4069-b513-3f6e776d2b15 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.829527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b58e604-89bf-4069-b513-3f6e776d2b15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.831177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b58e604-89bf-4069-b513-3f6e776d2b15 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.842888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 973/1961] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:24 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:24.842888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:INSERT=5,keystone:SELECT=95 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:24.845591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:24.856567 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76a7a74-8a9e-4cab-b00a-4d489f39251c service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:24.861155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:24.862031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.862311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.862584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] PATH_INFO: `/v3/projects/aa8ecbdfe94c442cbaf3daf8b422609b/users/9c3872da38ab434aba83fbfeb440e085/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.864609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa8ecbdfe94c442cbaf3daf8b422609b, user_id=9c3872da38ab434aba83fbfeb440e085, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:24.867298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:24.877998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 989/1962] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:24 2022] PUT /identity/v3/projects/aa8ecbdfe94c442cbaf3daf8b422609b/users/9c3872da38ab434aba83fbfeb440e085/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:24.888597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389eb2db-6e14-434e-bd1e-b4302e9be095 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:24.889159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389eb2db-6e14-434e-bd1e-b4302e9be095 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:24.889159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389eb2db-6e14-434e-bd1e-b4302e9be095 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:24.920886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-389eb2db-6e14-434e-bd1e-b4302e9be095 None None] MFA Rules not processed for user `9c3872da38ab434aba83fbfeb440e085`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:24.920886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-389eb2db-6e14-434e-bd1e-b4302e9be095 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:24.982445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:25.016521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 974/1963] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:24 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:25.028747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:25.091724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b7314b6-94c5-4534-aa33-d82f8c4b220e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:25.096680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:25.097454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:25.097736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:25.098006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:25.115082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:25.116807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:25.129102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 990/1964] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:25 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:25.207002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389eb2db-6e14-434e-bd1e-b4302e9be095 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:25.217756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389eb2db-6e14-434e-bd1e-b4302e9be095 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:25.221420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457ccea-0f68-48c3-907c-f330802fbc8e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:25.222076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457ccea-0f68-48c3-907c-f330802fbc8e service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:25.222360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457ccea-0f68-48c3-907c-f330802fbc8e service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:25.222647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457ccea-0f68-48c3-907c-f330802fbc8e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:25.287830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457ccea-0f68-48c3-907c-f330802fbc8e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:25.289362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457ccea-0f68-48c3-907c-f330802fbc8e service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:25.299477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 975/1965] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:25 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:25.391428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:25.452343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:25.456430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c4aa2f7-8e57-4b73-b12a-3ba733bf8559 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:25.460976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:25.461849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:25.462129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:25.462402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] PATH_INFO: `/v3/users/e41838459e404a8eb591ba43f82c8985` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:25.463759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e41838459e404a8eb591ba43f82c8985)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:25.466344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:25.662413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e41838459e404a8eb591ba43f82c8985'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:25.753441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e41838459e404a8eb591ba43f82c8985'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:25.761363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e41838459e404a8eb591ba43f82c8985'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:25.762045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e41838459e404a8eb591ba43f82c8985'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:25.762660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e41838459e404a8eb591ba43f82c8985'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:25.781411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 991/1966] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:25 2022] DELETE /identity/v3/users/e41838459e404a8eb591ba43f82c8985 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:25.785864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:25.786160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:25.786450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:25.804072 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:25.805277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:25.867556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 976/1967] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:25.880178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:25.881618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:25.940872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f6579-6a0e-4af6-8160-4de6b28c3094 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:25.944890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:25.945049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:25.945332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:25.945608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:25.955997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:25.957594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:25.968521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 992/1968] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:26.354386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:26.356807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:26.370998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376a3ac2-2d46-495d-931f-567ffa86bb47 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:26.377569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:26.378521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:26.378711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:26.379020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] PATH_INFO: `/v3/projects/785745b770024585902d82bdffdff518` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:26.380264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] RBAC: Authorizing `identity:delete_project(project_id=785745b770024585902d82bdffdff518)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:26.382195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:26.486302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:26.498607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '785745b770024585902d82bdffdff518', 'request_id': 'req-80d7f225-1a0a-472a-9ac4-edce8942ae60'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:26.504167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '785745b770024585902d82bdffdff518', 'request_id': 'req-80d7f225-1a0a-472a-9ac4-edce8942ae60'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:26.571097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '785745b770024585902d82bdffdff518', 'request_id': 'req-80d7f225-1a0a-472a-9ac4-edce8942ae60'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:26.583857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] The token cache is being invalidate because project 785745b770024585902d82bdffdff518 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:26.584145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:26.584776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:26.586345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 977/1969] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:26 2022] DELETE /identity/v3/projects/785745b770024585902d82bdffdff518 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:26.591161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:26.601059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c53ad77-62eb-41fd-9548-b46534ee838c service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:26.663042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:26.663931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:26.664216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:26.664488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] PATH_INFO: `/v3/users/5b18aec5f7a64c2399cb97e521b7c51b` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:26.665683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b18aec5f7a64c2399cb97e521b7c51b)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:26.667868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:26.690202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:26.917239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b18aec5f7a64c2399cb97e521b7c51b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.010561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b18aec5f7a64c2399cb97e521b7c51b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.021553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b18aec5f7a64c2399cb97e521b7c51b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.021553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b18aec5f7a64c2399cb97e521b7c51b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.021553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d6fe9533-5fe3-41a0-b4dc-0f53f364089b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b18aec5f7a64c2399cb97e521b7c51b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.039507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 993/1970] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:26 2022] DELETE /identity/v3/users/5b18aec5f7a64c2399cb97e521b7c51b => generated 0 bytes in 449 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:27.523412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:27.525635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:27.545672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80d7f225-1a0a-472a-9ac4-edce8942ae60 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:27.619222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:27.620098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:27.620383 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:27.620732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] PATH_INFO: `/v3/projects/ae363ceb53364d71bbf0831279347fbf` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:27.622585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae363ceb53364d71bbf0831279347fbf)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:27.624752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:27.755357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae363ceb53364d71bbf0831279347fbf', 'request_id': 'req-582eb432-b0fb-4579-891a-4bdbc19601b0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.761333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae363ceb53364d71bbf0831279347fbf', 'request_id': 'req-582eb432-b0fb-4579-891a-4bdbc19601b0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.845243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae363ceb53364d71bbf0831279347fbf', 'request_id': 'req-582eb432-b0fb-4579-891a-4bdbc19601b0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.858040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] The token cache is being invalidate because project ae363ceb53364d71bbf0831279347fbf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:27.858324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.858985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-582eb432-b0fb-4579-891a-4bdbc19601b0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:27.860542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 978/1971] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:27 2022] DELETE /identity/v3/projects/ae363ceb53364d71bbf0831279347fbf => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:27.865898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67adb1d6-d081-493a-9f2f-583cdc98129c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:27.866428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67adb1d6-d081-493a-9f2f-583cdc98129c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:27.866560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67adb1d6-d081-493a-9f2f-583cdc98129c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:27.884946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-67adb1d6-d081-493a-9f2f-583cdc98129c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:27.886146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-67adb1d6-d081-493a-9f2f-583cdc98129c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:27.946148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 994/1972] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:27.955243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba5532-1df3-48e6-adc8-daf75f3d08bb None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:27.955615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba5532-1df3-48e6-adc8-daf75f3d08bb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:27.956034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfba5532-1df3-48e6-adc8-daf75f3d08bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:27.979097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-bfba5532-1df3-48e6-adc8-daf75f3d08bb None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:27.981242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-bfba5532-1df3-48e6-adc8-daf75f3d08bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:27.993863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 979/1973] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:27.998267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67adb1d6-d081-493a-9f2f-583cdc98129c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.008209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67adb1d6-d081-493a-9f2f-583cdc98129c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.015724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:28.022184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.022184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.022184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.022184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:28.022184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:28.022184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:28.034095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 995/1974] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:28.039080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfba5532-1df3-48e6-adc8-daf75f3d08bb None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.052478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfba5532-1df3-48e6-adc8-daf75f3d08bb None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.055784 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:28.056556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.057276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.057276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.058126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:28.061097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:28.241550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 980/1975] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:28 2022] POST /identity/v3/projects => generated 383 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:28.245779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.256210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c6e9751-0d99-4544-877a-517851e3d9f8 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.325356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:28.325945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.326228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.326505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.327223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:28.329167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:28.398681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 996/1976] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:28 2022] POST /identity/v3/users => generated 292 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:28.403517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.413819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53ea5930-5caa-40cf-bb50-94ca50cc4b9c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.418432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:28.419383 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.419652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.419933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.420664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:28.422261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:28.432614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 981/1977] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:28.437346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.447947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3180a5-db89-44cc-8fa1-f8e7cfe23b6d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.453346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d944f4c4-8d64-427f-aa7a-698f28e1ac0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:28.454078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d944f4c4-8d64-427f-aa7a-698f28e1ac0d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.454383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d944f4c4-8d64-427f-aa7a-698f28e1ac0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.454662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d944f4c4-8d64-427f-aa7a-698f28e1ac0d admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.455485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d944f4c4-8d64-427f-aa7a-698f28e1ac0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:28.457161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d944f4c4-8d64-427f-aa7a-698f28e1ac0d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:28.469384 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 997/1978] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:28.473809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.486076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da99d332-58cd-4aa2-bf8e-b12761f92f11 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.491017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:28.491895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.492175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.492450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] PATH_INFO: `/v3/projects/4e1bac8093984013a2feb09f163b2ae7/users/52d5655225614c1abef25aae2ac8db06/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.501490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:28.508396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4e1bac8093984013a2feb09f163b2ae7, user_id=52d5655225614c1abef25aae2ac8db06, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:28.511068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:28.594467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 982/1979] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:28 2022] PUT /identity/v3/projects/4e1bac8093984013a2feb09f163b2ae7/users/52d5655225614c1abef25aae2ac8db06/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:28.605138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851d619-9615-4835-8c20-f10f836c5c03 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.605435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851d619-9615-4835-8c20-f10f836c5c03 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.605705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6851d619-9615-4835-8c20-f10f836c5c03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.613126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:28.635485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-6851d619-9615-4835-8c20-f10f836c5c03 None None] MFA Rules not processed for user `52d5655225614c1abef25aae2ac8db06`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:28.637359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6851d619-9615-4835-8c20-f10f836c5c03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:28.726549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 998/1980] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:28 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:28.731197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.804040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6a54ed-34b6-4287-9866-1d22da3c73d0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.808704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:28.809374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:28.809691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:28.810000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:28.810711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:28.812716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:28.927429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 983/1981] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:28 2022] POST /identity/v3/projects => generated 383 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:28.931355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6851d619-9615-4835-8c20-f10f836c5c03 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:28.942141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6851d619-9615-4835-8c20-f10f836c5c03 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:28.999631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:29.000307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:29.000591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:29.000865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:29.001565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:29.003569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:29.102836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 999/1982] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:28 2022] POST /identity/v3/users => generated 298 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:29.106699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:29.119067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80f7067-11a3-4751-a3a9-0f409ab2b623 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:29.127397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:29.128179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:29.128540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:29.129080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:29.130049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:29.132505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:29.165495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 984/1983] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 59 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:29.170652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:29.186415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d81758b-ada5-4769-9fb3-fbe8ccc9101c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:29.197376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:29.198336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:29.198615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:29.199736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] PATH_INFO: `/v3/projects/4ce2cd286b2c41b89ab361aa3bc2dd18/users/278de9e7dbe34a96ac56ecdd4c86941d/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:29.231411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ce2cd286b2c41b89ab361aa3bc2dd18, user_id=278de9e7dbe34a96ac56ecdd4c86941d, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:29.234865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:29.371105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1000/1984] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:29 2022] PUT /identity/v3/projects/4ce2cd286b2c41b89ab361aa3bc2dd18/users/278de9e7dbe34a96ac56ecdd4c86941d/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:29.375363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:29.385944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dbe90d4-eeb3-4bfd-bc6e-62f37acb5452 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:29.444654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02fe4fd-681f-4077-9306-361e5e151b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:29.444788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fe4fd-681f-4077-9306-361e5e151b31 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:29.445033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fe4fd-681f-4077-9306-361e5e151b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:29.445311 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02fe4fd-681f-4077-9306-361e5e151b31 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:29.446053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02fe4fd-681f-4077-9306-361e5e151b31 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:29.447707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02fe4fd-681f-4077-9306-361e5e151b31 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:29.458484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 985/1985] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:29 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:29.462694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:29.473520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eafddab-5d08-45fd-a9e9-b7b2f462d12f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:29.478348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:29.479273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:29.479550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:29.479826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] PATH_INFO: `/v3/projects/4ce2cd286b2c41b89ab361aa3bc2dd18/users/278de9e7dbe34a96ac56ecdd4c86941d/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:29.482048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4ce2cd286b2c41b89ab361aa3bc2dd18, user_id=278de9e7dbe34a96ac56ecdd4c86941d, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:29.484809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:29.496943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1001/1986] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:29 2022] PUT /identity/v3/projects/4ce2cd286b2c41b89ab361aa3bc2dd18/users/278de9e7dbe34a96ac56ecdd4c86941d/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:29.507075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:29.507367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:29.507680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:29.546341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] MFA Rules not processed for user `278de9e7dbe34a96ac56ecdd4c86941d`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:29.546341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:29.593871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:29.619478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 986/1987] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:29 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:29.644387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:29.646189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:29.702979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:29.708233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0559eab0-d06e-4c4e-b92f-feed25076651 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:29.717864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-956f73a2-2ba0-4b11-b831-947e1e791ef4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:29.719626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956f73a2-2ba0-4b11-b831-947e1e791ef4 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:29.719626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956f73a2-2ba0-4b11-b831-947e1e791ef4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:29.719626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956f73a2-2ba0-4b11-b831-947e1e791ef4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:29.790399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956f73a2-2ba0-4b11-b831-947e1e791ef4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:29.792343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-956f73a2-2ba0-4b11-b831-947e1e791ef4 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:29.803671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1002/1988] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:30.136968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:30.138888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:30.222592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-493c0368-8008-4a02-a74c-b698cb255225 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:30.227886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:30.231138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:30.231587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:30.231955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] PATH_INFO: `/v3/users/2212c1f9c30045efabb132144cb1f1e0` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:30.233833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=2212c1f9c30045efabb132144cb1f1e0)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:30.235371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:30.477994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2212c1f9c30045efabb132144cb1f1e0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:30.565821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2212c1f9c30045efabb132144cb1f1e0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:30.566683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2212c1f9c30045efabb132144cb1f1e0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:30.577210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2212c1f9c30045efabb132144cb1f1e0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:30.578102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2212c1f9c30045efabb132144cb1f1e0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:30.598444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 987/1989] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:30 2022] DELETE /identity/v3/users/2212c1f9c30045efabb132144cb1f1e0 => generated 0 bytes in 462 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:30.603027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:30.603547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:30.603638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:30.621184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:30.622402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:30.698239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1003/1990] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:30.708583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:30.710670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:30.784148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-462c2fcd-7c6b-4ee5-ba82-2db362aed1fa admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:30.788091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:30.788755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:30.789047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:30.789327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:30.800080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:30.801524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:30.820527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 988/1991] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:31.582795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:31.584380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:31.609633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3462c8af-6258-486b-bc0c-3b39ef5402ea None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:31.618005 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:31.630242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:31.630564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:31.630871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] PATH_INFO: `/v3/projects/286fd5b29ea94d898673abfb9d029fa7` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:31.632587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] RBAC: Authorizing `identity:delete_project(project_id=286fd5b29ea94d898673abfb9d029fa7)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:31.635147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:31.780207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '286fd5b29ea94d898673abfb9d029fa7', 'request_id': 'req-c0762420-60ca-4ee3-a953-a2174d904591'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:31.874061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '286fd5b29ea94d898673abfb9d029fa7', 'request_id': 'req-c0762420-60ca-4ee3-a953-a2174d904591'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:31.881254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '286fd5b29ea94d898673abfb9d029fa7', 'request_id': 'req-c0762420-60ca-4ee3-a953-a2174d904591'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:31.892195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] The token cache is being invalidate because project 286fd5b29ea94d898673abfb9d029fa7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:31.892509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:31.893160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:31.894741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1004/1992] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:31 2022] DELETE /identity/v3/projects/286fd5b29ea94d898673abfb9d029fa7 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:31.898739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:31.905486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:31.908758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e698ec30-e840-4dd8-a445-c7aef925942c service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:31.967481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:31.968373 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:31.968660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:31.968944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] PATH_INFO: `/v3/users/9c3872da38ab434aba83fbfeb440e085` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:31.969137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:31.970182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9c3872da38ab434aba83fbfeb440e085)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:31.970479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:31.972182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:31.983347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0762420-60ca-4ee3-a953-a2174d904591 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:31.987962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:31.988835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:31.989118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:31.989396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] PATH_INFO: `/v3/users/361a4f34e0b0435e944b6fe1ab485a1a` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:31.990641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] RBAC: Authorizing `identity:delete_user(user_id=361a4f34e0b0435e944b6fe1ab485a1a)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:31.992587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:32.014408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:32.162107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9c3872da38ab434aba83fbfeb440e085'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.186430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '361a4f34e0b0435e944b6fe1ab485a1a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.245048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9c3872da38ab434aba83fbfeb440e085'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.246412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9c3872da38ab434aba83fbfeb440e085'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.253911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9c3872da38ab434aba83fbfeb440e085'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.254634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0bde42a7-c5b0-4b52-a83f-b98a16d00e06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9c3872da38ab434aba83fbfeb440e085'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.272870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 989/1993] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:31 2022] DELETE /identity/v3/users/9c3872da38ab434aba83fbfeb440e085 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:32.399728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '361a4f34e0b0435e944b6fe1ab485a1a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.407118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '361a4f34e0b0435e944b6fe1ab485a1a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.407802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '361a4f34e0b0435e944b6fe1ab485a1a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.408418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '361a4f34e0b0435e944b6fe1ab485a1a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.425354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1005/1994] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:31 2022] DELETE /identity/v3/users/361a4f34e0b0435e944b6fe1ab485a1a => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:32.430027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:32.433733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:32.433733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:32.448215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:32.450247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:32.518595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 990/1995] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:32.528782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:32.530358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:32.590388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ee8bc22-ea54-4ce6-b21c-2c8db3952f23 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:32.600225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0534b722-b588-4359-a740-7caf171bcd80 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:32.601143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534b722-b588-4359-a740-7caf171bcd80 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:32.601424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534b722-b588-4359-a740-7caf171bcd80 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:32.601722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0534b722-b588-4359-a740-7caf171bcd80 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:32.612224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0534b722-b588-4359-a740-7caf171bcd80 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:32.613666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0534b722-b588-4359-a740-7caf171bcd80 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:32.624590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1006/1996] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:32.698292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:32.700390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:32.736456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad54051-f5e5-41fd-9fd6-0d1b3f3966f1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:32.741715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:32.742715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:32.742926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:32.743180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] PATH_INFO: `/v3/projects/aa8ecbdfe94c442cbaf3daf8b422609b` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:32.744561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa8ecbdfe94c442cbaf3daf8b422609b)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:32.748075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:32.775281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:32.870228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa8ecbdfe94c442cbaf3daf8b422609b', 'request_id': 'req-36b92a5f-8896-4a16-bb8c-b40997356e75'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.875747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa8ecbdfe94c442cbaf3daf8b422609b', 'request_id': 'req-36b92a5f-8896-4a16-bb8c-b40997356e75'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:32.987511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa8ecbdfe94c442cbaf3daf8b422609b', 'request_id': 'req-36b92a5f-8896-4a16-bb8c-b40997356e75'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.004250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] The token cache is being invalidate because project aa8ecbdfe94c442cbaf3daf8b422609b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:33.008642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.008642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-36b92a5f-8896-4a16-bb8c-b40997356e75 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.009187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 991/1997] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:32 2022] DELETE /identity/v3/projects/aa8ecbdfe94c442cbaf3daf8b422609b => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:33.015040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c2ec0-d791-42cc-82e5-5126282847ce None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.015557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c2ec0-d791-42cc-82e5-5126282847ce None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.017550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c4c2ec0-d791-42cc-82e5-5126282847ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.038231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0c4c2ec0-d791-42cc-82e5-5126282847ce None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:33.039513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0c4c2ec0-d791-42cc-82e5-5126282847ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:33.123181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1007/1998] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:33.131162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d21aaf-ed5b-497d-8462-039a8e2878a9 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.131555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d21aaf-ed5b-497d-8462-039a8e2878a9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.131902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d21aaf-ed5b-497d-8462-039a8e2878a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.152384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f6d21aaf-ed5b-497d-8462-039a8e2878a9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:33.153897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f6d21aaf-ed5b-497d-8462-039a8e2878a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:33.166581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 992/1999] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:33.171535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4c2ec0-d791-42cc-82e5-5126282847ce None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.181611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c4c2ec0-d791-42cc-82e5-5126282847ce None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.186930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.187649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.187942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.188222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.189092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:33.189407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.191039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.212327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1008/2000] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:33.217558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d21aaf-ed5b-497d-8462-039a8e2878a9 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.226532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d21aaf-ed5b-497d-8462-039a8e2878a9 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.231249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.231899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.232179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.232452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.233153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.235077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.333133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.337213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:33.354153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b98e8f46-8195-404b-a0bb-983b851eb037 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.361609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.363207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.363465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.363865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] PATH_INFO: `/v3/projects/452717ce7f314047b3664d11fd81be3a` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.365116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] RBAC: Authorizing `identity:delete_project(project_id=452717ce7f314047b3664d11fd81be3a)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.367789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.388496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 993/2001] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:33 2022] POST /identity/v3/projects => generated 359 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:33.389820 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.401457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=8,keystone:SELECT=91,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:33.493201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc822520-7a38-4099-a802-1b115df5a73f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.500613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.501710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.502052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.502461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.503480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.505832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.521671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '452717ce7f314047b3664d11fd81be3a', 'request_id': 'req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.629761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 994/2002] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:33 2022] POST /identity/v3/users => generated 280 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:33.630279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '452717ce7f314047b3664d11fd81be3a', 'request_id': 'req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.634405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.654561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '452717ce7f314047b3664d11fd81be3a', 'request_id': 'req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.662734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd6eb57-bdca-4868-948e-49cc5a3c7002 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.669360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] The token cache is being invalidate because project 452717ce7f314047b3664d11fd81be3a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:33.670368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.671662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.674303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1009/2003] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:33 2022] DELETE /identity/v3/projects/452717ce7f314047b3664d11fd81be3a => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:33.678665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.680719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:33.739762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.740755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.741128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.741468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.742372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.744220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.756015 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 995/2004] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:33 2022] GET /identity/v3/roles => generated 1420 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:33.759671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.762172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:33.768845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a3b91d-80ee-4e7d-86cd-18fb5c9fd87e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.772699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:33.774678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.775674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.775954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.778662 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] PATH_INFO: `/v3/users/47e6a08ab9584b3abaf3becf7c39e0b9` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.778852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a56b5afd-4e6d-4c2b-b065-1baffc5aafa0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.778960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47e6a08ab9584b3abaf3becf7c39e0b9)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.780212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.783814 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.784568 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.785366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.785766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.786908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.788851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.804588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 996/2005] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:33 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:33.809890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:33.825886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fc65622-32c5-4033-8201-70c0d48d50bd admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:33.831547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ead42d-6b9d-4b61-9c68-9d62e983b591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:33.832445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ead42d-6b9d-4b61-9c68-9d62e983b591 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.832729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ead42d-6b9d-4b61-9c68-9d62e983b591 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.832999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ead42d-6b9d-4b61-9c68-9d62e983b591 admin admin] PATH_INFO: `/v3/projects/75bfc0dda3e84785bd706d753d7b4047/users/2ea6096dd6da43b4946d09634f9c43ca/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.849007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ead42d-6b9d-4b61-9c68-9d62e983b591 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75bfc0dda3e84785bd706d753d7b4047, user_id=2ea6096dd6da43b4946d09634f9c43ca, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:33.851714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ead42d-6b9d-4b61-9c68-9d62e983b591 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:33.915441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 997/2006] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:33 2022] PUT /identity/v3/projects/75bfc0dda3e84785bd706d753d7b4047/users/2ea6096dd6da43b4946d09634f9c43ca/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:33.926479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:33.926804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:33.927130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:33.951018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47e6a08ab9584b3abaf3becf7c39e0b9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:33.963777 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] MFA Rules not processed for user `2ea6096dd6da43b4946d09634f9c43ca`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:33.965269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:34.045496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47e6a08ab9584b3abaf3becf7c39e0b9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.047828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 998/2007] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:33 2022] POST /identity/v3/auth/tokens => generated 3226 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:34.068249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47e6a08ab9584b3abaf3becf7c39e0b9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.069260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47e6a08ab9584b3abaf3becf7c39e0b9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.070444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47e6a08ab9584b3abaf3becf7c39e0b9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.091020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:34.096388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:34.110034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1010/2008] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:33 2022] DELETE /identity/v3/users/47e6a08ab9584b3abaf3becf7c39e0b9 => generated 0 bytes in 432 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:34.188368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca300bf-70bc-49ad-b135-1d09d4e7e19d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:34.256545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:34.257206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:34.257489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:34.257767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:34.259735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:34.292220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:34.324107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:34.325757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:34.335579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 999/2009] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:34 2022] GET /identity/v3/auth/tokens => generated 3226 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:19:34.350858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:34.352451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:34.399923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=17,keystone:SELECT=81,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:34.443574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c871ec55-f577-46d7-9dde-37e45f430791 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:34.448543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:34.449755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:34.450039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:34.450314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] PATH_INFO: `/v3/users/2ea6096dd6da43b4946d09634f9c43ca` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:34.451558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ea6096dd6da43b4946d09634f9c43ca)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:34.453762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:34.584511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:34.585973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:34.613637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15cf2edd-bfe8-49cd-b025-6c034c72b1ab service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:34.619199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:34.622804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:34.622804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:34.622804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] PATH_INFO: `/v3/projects/bbad5980e6ad45aba5b72ff0b01d5c6b` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:34.622804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bbad5980e6ad45aba5b72ff0b01d5c6b)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:34.624003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:34.662893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ea6096dd6da43b4946d09634f9c43ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.727742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ea6096dd6da43b4946d09634f9c43ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.740232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ea6096dd6da43b4946d09634f9c43ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.741153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ea6096dd6da43b4946d09634f9c43ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.742119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67c44897-4458-4cd7-8e16-9be35f4af2b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ea6096dd6da43b4946d09634f9c43ca'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.749674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bbad5980e6ad45aba5b72ff0b01d5c6b', 'request_id': 'req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.762473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bbad5980e6ad45aba5b72ff0b01d5c6b', 'request_id': 'req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.763084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1011/2010] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:34 2022] DELETE /identity/v3/users/2ea6096dd6da43b4946d09634f9c43ca => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:34.766133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:34.766486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:34.766801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:34.785430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:34.786779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:34.856024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1012/2011] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:34.872318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:34.875924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:34.934523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:34.938463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4db9f0a-878c-4aa0-888b-f78142475a76 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:34.942208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84bcbf9-a619-400d-a68a-05c515e4ae1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:34.942958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bcbf9-a619-400d-a68a-05c515e4ae1b service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:34.943245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bcbf9-a619-400d-a68a-05c515e4ae1b service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:34.943525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bcbf9-a619-400d-a68a-05c515e4ae1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:34.953330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84bcbf9-a619-400d-a68a-05c515e4ae1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:34.954788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84bcbf9-a619-400d-a68a-05c515e4ae1b service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:34.964360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1013/2012] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:34.970551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bbad5980e6ad45aba5b72ff0b01d5c6b', 'request_id': 'req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.984714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] The token cache is being invalidate because project bbad5980e6ad45aba5b72ff0b01d5c6b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:34.985121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.985896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d844ea28-8b04-4ca3-a81c-c5ac0117e5e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:34.988002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1000/2013] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:34 2022] DELETE /identity/v3/projects/bbad5980e6ad45aba5b72ff0b01d5c6b => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:34.992470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47894464-cd15-4020-9edb-7e62cea2c1a8 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:34.992825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47894464-cd15-4020-9edb-7e62cea2c1a8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:34.993191 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47894464-cd15-4020-9edb-7e62cea2c1a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.011960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-47894464-cd15-4020-9edb-7e62cea2c1a8 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:35.013163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-47894464-cd15-4020-9edb-7e62cea2c1a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:35.021403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1014/2014] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:35.029414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca63487-051d-4e8d-b2e6-cfe8d2d10aab None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.029719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca63487-051d-4e8d-b2e6-cfe8d2d10aab None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.030003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ca63487-051d-4e8d-b2e6-cfe8d2d10aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.071849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2ca63487-051d-4e8d-b2e6-cfe8d2d10aab None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:35.073595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2ca63487-051d-4e8d-b2e6-cfe8d2d10aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:35.081135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1001/2015] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:35.085517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47894464-cd15-4020-9edb-7e62cea2c1a8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.095401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47894464-cd15-4020-9edb-7e62cea2c1a8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:35.100017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:35.102367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.102367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.102367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.102367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:35.102367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:35.103773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:35.115681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1015/2016] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:35.119962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca63487-051d-4e8d-b2e6-cfe8d2d10aab None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.130196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ca63487-051d-4e8d-b2e6-cfe8d2d10aab None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:35.135583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:35.136272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.136704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.137002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.137715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:35.139704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:35.287678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1002/2017] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:35 2022] POST /identity/v3/projects => generated 355 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:35.292010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.313966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dec9304f-9bcf-44c1-8d0a-0db73ad57c3d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:35.466633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:35.467450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.467730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.468027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.468788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:35.470691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:35.535314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.538875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:35.559569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a8cab1-c52f-44c2-b0da-63c7bffa75ec admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:35.565565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:35.566440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.566726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.567037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] PATH_INFO: `/v3/projects/75bfc0dda3e84785bd706d753d7b4047` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.568397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75bfc0dda3e84785bd706d753d7b4047)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:35.570357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:35.586495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1016/2018] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:35 2022] POST /identity/v3/users => generated 278 bytes in 293 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:35.588897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.612727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-248e1997-fad3-410b-affb-849adfebff9d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:35.617857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:35.618582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.618900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.619179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.619945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:35.622669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:35.633744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1017/2019] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:35.638177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.648383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19328c08-d7e4-423c-921b-c79aa197fa38 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:35.653577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:35.654246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.654528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.654829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.655558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:35.657151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:35.668209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1018/2020] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:35 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:35.672483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.681880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e680961-062f-45dc-88ec-827d8498d5ce admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:35.687396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bf83ee-e8b8-4792-9031-c2fccc784712 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:35.688274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf83ee-e8b8-4792-9031-c2fccc784712 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.688560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf83ee-e8b8-4792-9031-c2fccc784712 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.688846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bf83ee-e8b8-4792-9031-c2fccc784712 admin admin] PATH_INFO: `/v3/projects/636e780d917d4b96bbafd985d415997e/users/c546618fc28d40e9aa015a4ba8482351/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.699919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75bfc0dda3e84785bd706d753d7b4047', 'request_id': 'req-9336c970-3b55-45a4-8512-23d8d09da052'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:35.704271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bf83ee-e8b8-4792-9031-c2fccc784712 admin admin] RBAC: Authorizing `identity:create_grant(project_id=636e780d917d4b96bbafd985d415997e, user_id=c546618fc28d40e9aa015a4ba8482351, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:35.709129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75bfc0dda3e84785bd706d753d7b4047', 'request_id': 'req-9336c970-3b55-45a4-8512-23d8d09da052'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:35.709305 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bf83ee-e8b8-4792-9031-c2fccc784712 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:35.788219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75bfc0dda3e84785bd706d753d7b4047', 'request_id': 'req-9336c970-3b55-45a4-8512-23d8d09da052'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:35.801084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1019/2021] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:35 2022] PUT /identity/v3/projects/636e780d917d4b96bbafd985d415997e/users/c546618fc28d40e9aa015a4ba8482351/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:35.801919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] The token cache is being invalidate because project 75bfc0dda3e84785bd706d753d7b4047 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:35.802227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:35.802886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:35.804465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1003/2022] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:35 2022] DELETE /identity/v3/projects/75bfc0dda3e84785bd706d753d7b4047 => generated 0 bytes in 271 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:35.812522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a65149-7522-4005-951a-c4f857d60da8 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:35.812835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a65149-7522-4005-951a-c4f857d60da8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:35.813215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30a65149-7522-4005-951a-c4f857d60da8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:35.861192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-30a65149-7522-4005-951a-c4f857d60da8 None None] MFA Rules not processed for user `c546618fc28d40e9aa015a4ba8482351`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:35.862601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-30a65149-7522-4005-951a-c4f857d60da8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:35.957884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1020/2023] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:35 2022] POST /identity/v3/auth/tokens => generated 3222 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:35.962530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:35.965224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:35.975619 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:36.041894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9336c970-3b55-45a4-8512-23d8d09da052 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:36.100528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:36.101229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.101598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.101884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.102594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:36.104630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:36.166025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1004/2024] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:35 2022] POST /identity/v3/projects => generated 357 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:36.170941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a65149-7522-4005-951a-c4f857d60da8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:36.177592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:36.181953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30a65149-7522-4005-951a-c4f857d60da8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:36.240521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:36.241408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.241689 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.241962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.242660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:36.244821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:36.308084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1021/2025] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:36 2022] POST /identity/v3/users => generated 285 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:36.312137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:36.322805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-232f7a6a-c076-496d-baa8-537a1a996d19 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:36.328282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:36.328986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.329277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.329559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.330383 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:36.332008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:36.343465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1005/2026] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:36.347519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:36.357989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b131e192-a6cc-49d7-8185-f52bca3a1560 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:36.362705 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:36.363612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.363898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.364205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] PATH_INFO: `/v3/projects/34fe7c1656d844e9b920116305e053b6/users/af701bcd5f0c471384344cd671e3b2fe/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.379023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] RBAC: Authorizing `identity:create_grant(project_id=34fe7c1656d844e9b920116305e053b6, user_id=af701bcd5f0c471384344cd671e3b2fe, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:36.381688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:36.440104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1022/2027] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:36 2022] PUT /identity/v3/projects/34fe7c1656d844e9b920116305e053b6/users/af701bcd5f0c471384344cd671e3b2fe/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:36.454747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:36.467717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbd2e93-0bbd-4549-9b64-2273feb56e98 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:36.539130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:36.540901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:36.592852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:36.594958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.595458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.595834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.596789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:36.598631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:36.619131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1006/2028] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:36 2022] GET /identity/v3/roles => generated 1420 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:36.619228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4efd9b-aec6-4274-b4b7-fe4753cd779b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:36.619346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:36.623420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:36.623420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.623420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.623420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.624086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:36.633024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d224c712-94b8-4350-83aa-50732755955c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:36.642666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf7114a-38ee-41be-9d84-9e65f8ef7684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:36.644470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf7114a-38ee-41be-9d84-9e65f8ef7684 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.644766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf7114a-38ee-41be-9d84-9e65f8ef7684 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.645050 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf7114a-38ee-41be-9d84-9e65f8ef7684 admin admin] PATH_INFO: `/v3/projects/34fe7c1656d844e9b920116305e053b6/users/af701bcd5f0c471384344cd671e3b2fe/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.647863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf7114a-38ee-41be-9d84-9e65f8ef7684 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34fe7c1656d844e9b920116305e053b6, user_id=af701bcd5f0c471384344cd671e3b2fe, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:36.650523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf7114a-38ee-41be-9d84-9e65f8ef7684 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:36.665508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1007/2029] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:36 2022] PUT /identity/v3/projects/34fe7c1656d844e9b920116305e053b6/users/af701bcd5f0c471384344cd671e3b2fe/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:36.677193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4350e43d-e50f-47f1-8443-2bfb067852c6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:36.677756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4350e43d-e50f-47f1-8443-2bfb067852c6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:36.678035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4350e43d-e50f-47f1-8443-2bfb067852c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:36.706427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:36.707943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:36.708234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:36.717324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4350e43d-e50f-47f1-8443-2bfb067852c6 None None] MFA Rules not processed for user `af701bcd5f0c471384344cd671e3b2fe`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:36.718594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4350e43d-e50f-47f1-8443-2bfb067852c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:36.724118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1023/2030] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:36 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:19:36.802665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1008/2031] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:36 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:37.275528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:37.277210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:37.295547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d068ac-6f33-424c-a16b-d79186720950 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:37.315227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:37.348324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:37.349119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:37.349399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:37.349751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:37.360571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:37.362079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:37.372985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1024/2032] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:37 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:37.663100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4350e43d-e50f-47f1-8443-2bfb067852c6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:37.673634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4350e43d-e50f-47f1-8443-2bfb067852c6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:37.677157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eec135df-e952-46ca-b7e9-306ac79e4651 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:37.677805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec135df-e952-46ca-b7e9-306ac79e4651 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:37.678085 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec135df-e952-46ca-b7e9-306ac79e4651 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:37.678362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec135df-e952-46ca-b7e9-306ac79e4651 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:37.736146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec135df-e952-46ca-b7e9-306ac79e4651 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:37.737613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eec135df-e952-46ca-b7e9-306ac79e4651 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:37.747165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1009/2033] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:37 2022] GET /identity/v3/auth/tokens => generated 3222 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:42.690359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:42.695680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:42.808695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec50cc7-039c-4841-b39c-771c527ee421 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:42.814729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:42.815834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:42.816634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:42.816975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] PATH_INFO: `/v3/users/52d5655225614c1abef25aae2ac8db06` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:42.818292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52d5655225614c1abef25aae2ac8db06)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:42.820317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:43.000996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52d5655225614c1abef25aae2ac8db06'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.045236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52d5655225614c1abef25aae2ac8db06'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.054261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52d5655225614c1abef25aae2ac8db06'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.059073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52d5655225614c1abef25aae2ac8db06'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.059721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52d5655225614c1abef25aae2ac8db06'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.086058 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1025/2034] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:42 2022] DELETE /identity/v3/users/52d5655225614c1abef25aae2ac8db06 => generated 0 bytes in 396 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:43.090464 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:43.091437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:43.091924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:43.113336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:43.114916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:43.185855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1010/2035] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:43 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:43.195397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:43.197816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:43.269971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:43.287686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-374b63af-0627-4eb2-90be-cb80c51e7764 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:43.296514 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-520bc65f-e834-4b1e-9023-71f036bdae01 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:43.297188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520bc65f-e834-4b1e-9023-71f036bdae01 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:43.297501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520bc65f-e834-4b1e-9023-71f036bdae01 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:43.297781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520bc65f-e834-4b1e-9023-71f036bdae01 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:43.309269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520bc65f-e834-4b1e-9023-71f036bdae01 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:43.312801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-520bc65f-e834-4b1e-9023-71f036bdae01 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:43.320946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1026/2036] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:43.400757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:43.402743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:43.437755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd54966-1d05-40b5-9839-3f7154309562 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:43.442296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:43.443204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:43.443491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:43.443766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] PATH_INFO: `/v3/users/a0a6364a4a884b09b5eb80897c19e296` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:43.444949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a0a6364a4a884b09b5eb80897c19e296)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:43.447145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:43.468413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01a3e6-900a-410a-8a1a-601f6577726f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:43.468914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01a3e6-900a-410a-8a1a-601f6577726f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:43.469048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01a3e6-900a-410a-8a1a-601f6577726f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:43.487704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8b01a3e6-900a-410a-8a1a-601f6577726f None None] MFA Rules not processed for user `3ccee18a56e44ccebfa14d264deb219b`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:43.490401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8b01a3e6-900a-410a-8a1a-601f6577726f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:43.551442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1027/2037] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:43 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:43.628905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:43.643566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a0a6364a4a884b09b5eb80897c19e296'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.696133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a0a6364a4a884b09b5eb80897c19e296'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.696749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a0a6364a4a884b09b5eb80897c19e296'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.708741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a0a6364a4a884b09b5eb80897c19e296'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.709469 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a0a6364a4a884b09b5eb80897c19e296'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:43.741724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1011/2038] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:43 2022] DELETE /identity/v3/users/a0a6364a4a884b09b5eb80897c19e296 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:44.069902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01a3e6-900a-410a-8a1a-601f6577726f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:44.083030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01a3e6-900a-410a-8a1a-601f6577726f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:44.093730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:44.095372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:44.157792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:44.158899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.163238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.163238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] PATH_INFO: `/v3/projects/1500083e98d84701b246fbbd749354fa` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.163238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1500083e98d84701b246fbbd749354fa)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:44.165037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:44.183555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66834425-e853-48ff-bb5a-06a43b496c19 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:44.190317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:44.191472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.191968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.192505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] PATH_INFO: `/v3/projects/4e1bac8093984013a2feb09f163b2ae7` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.194175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4e1bac8093984013a2feb09f163b2ae7)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:44.197869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:44.312786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1500083e98d84701b246fbbd749354fa', 'request_id': 'req-494a5053-a87d-4008-9520-f8649ca4fa57'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.350531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4e1bac8093984013a2feb09f163b2ae7', 'request_id': 'req-a38635cf-c685-442c-900f-8249e18f9af6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.365535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4e1bac8093984013a2feb09f163b2ae7', 'request_id': 'req-a38635cf-c685-442c-900f-8249e18f9af6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.387143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1500083e98d84701b246fbbd749354fa', 'request_id': 'req-494a5053-a87d-4008-9520-f8649ca4fa57'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.394839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1500083e98d84701b246fbbd749354fa', 'request_id': 'req-494a5053-a87d-4008-9520-f8649ca4fa57'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.408792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] The token cache is being invalidate because project 1500083e98d84701b246fbbd749354fa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:44.410635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.411439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-494a5053-a87d-4008-9520-f8649ca4fa57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.413365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1028/2039] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:44 2022] DELETE /identity/v3/projects/1500083e98d84701b246fbbd749354fa => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:44.419094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bec69-0c0f-46b0-a7c3-1e92ab9b7258 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.420051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bec69-0c0f-46b0-a7c3-1e92ab9b7258 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.420169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bec69-0c0f-46b0-a7c3-1e92ab9b7258 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.439958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-782bec69-0c0f-46b0-a7c3-1e92ab9b7258 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:44.441194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-782bec69-0c0f-46b0-a7c3-1e92ab9b7258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:44.454557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:44.490650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4e1bac8093984013a2feb09f163b2ae7', 'request_id': 'req-a38635cf-c685-442c-900f-8249e18f9af6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.506129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] The token cache is being invalidate because project 4e1bac8093984013a2feb09f163b2ae7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:44.506430 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.507354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.510134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1012/2040] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:44 2022] DELETE /identity/v3/projects/4e1bac8093984013a2feb09f163b2ae7 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:44.513834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:44.518515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:44.543071 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1029/2041] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:44.555019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9aa939-85f5-4366-89b4-b9a0a9195efd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.555019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9aa939-85f5-4366-89b4-b9a0a9195efd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.555019 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9aa939-85f5-4366-89b4-b9a0a9195efd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.595173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ca9aa939-85f5-4366-89b4-b9a0a9195efd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:44.595173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ca9aa939-85f5-4366-89b4-b9a0a9195efd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:44.595339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:44.607894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1030/2042] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:44 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:44.608860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9aa939-85f5-4366-89b4-b9a0a9195efd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:44.618509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9aa939-85f5-4366-89b4-b9a0a9195efd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:44.622201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a38635cf-c685-442c-900f-8249e18f9af6 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:44.623898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:44.624661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.625010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.625355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.626160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:44.626533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:44.628113 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:44.628453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:44.629475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.629942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.630370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] PATH_INFO: `/v3/users/278de9e7dbe34a96ac56ecdd4c86941d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.632518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] RBAC: Authorizing `identity:delete_user(user_id=278de9e7dbe34a96ac56ecdd4c86941d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:44.635293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:44.642826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1031/2043] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:44.646905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:44.657451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b66d197e-f9d1-4306-b9fa-6906426c8f17 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:44.663100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:44.663940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.664120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.664447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.665131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:44.667195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:44.793850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1032/2044] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:44 2022] POST /identity/v3/projects => generated 387 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:44.795260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:44.858670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6f09581-376a-41a6-b042-d0a6da437860 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:44.863575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:44.863818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '278de9e7dbe34a96ac56ecdd4c86941d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.864341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:44.864628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:44.864905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:44.865644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:44.867627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:44.936148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '278de9e7dbe34a96ac56ecdd4c86941d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.936879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '278de9e7dbe34a96ac56ecdd4c86941d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.944851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '278de9e7dbe34a96ac56ecdd4c86941d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.945512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '278de9e7dbe34a96ac56ecdd4c86941d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:44.963016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1033/2045] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:44 2022] POST /identity/v3/users => generated 294 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:44.964159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:44.966014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:44.967232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1013/2046] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:44 2022] DELETE /identity/v3/users/278de9e7dbe34a96ac56ecdd4c86941d => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:44.978433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8989424-5bf4-479b-8e98-eaf6c83fb23d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:45.053653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:45.054440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:45.054725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:45.055047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:45.055857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:45.058520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:45.070551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1034/2047] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:44 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:45.075492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:45.091074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54a8942-d234-45c3-956d-9595e299f655 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:45.096923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b89f94-3249-48ce-acaf-24d1b6e2422f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:45.098176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b89f94-3249-48ce-acaf-24d1b6e2422f admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:45.098467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b89f94-3249-48ce-acaf-24d1b6e2422f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:45.098746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b89f94-3249-48ce-acaf-24d1b6e2422f admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:45.099538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b89f94-3249-48ce-acaf-24d1b6e2422f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:45.101242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b89f94-3249-48ce-acaf-24d1b6e2422f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:45.113695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1014/2048] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:45 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:45.118130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:45.128845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bda56a4-28cb-45bc-9017-8228e8ea5369 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:45.134144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:45.135061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:45.135338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:45.135610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] PATH_INFO: `/v3/projects/51617cc935204a6cba46c6d4d6a24cb3/users/15e708082e294dd9a0834ef21ea46fd9/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:45.156275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] RBAC: Authorizing `identity:create_grant(project_id=51617cc935204a6cba46c6d4d6a24cb3, user_id=15e708082e294dd9a0834ef21ea46fd9, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:45.159236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:45.205587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:45.237703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1035/2049] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:45 2022] PUT /identity/v3/projects/51617cc935204a6cba46c6d4d6a24cb3/users/15e708082e294dd9a0834ef21ea46fd9/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:45.242574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:45.242908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:45.243402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:45.262751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:45.264045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:45.324339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1015/2050] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:45.333935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:45.335821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:45.393696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e26833-71d0-4027-9b83-d774fe16e583 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:45.397101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e75b6861-121f-4990-91b8-10d57ea9f363 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:45.397759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b6861-121f-4990-91b8-10d57ea9f363 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:45.398044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b6861-121f-4990-91b8-10d57ea9f363 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:45.398325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e75b6861-121f-4990-91b8-10d57ea9f363 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:45.408123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b6861-121f-4990-91b8-10d57ea9f363 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:45.409660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e75b6861-121f-4990-91b8-10d57ea9f363 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:45.420400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1036/2051] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:45 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:19:45.504578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:45.507362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:45.523535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adf4d293-5d57-4e8d-9149-9d0cc1eff246 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:45.536667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:45.538698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:45.538941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:45.539730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] PATH_INFO: `/v3/projects/4ce2cd286b2c41b89ab361aa3bc2dd18` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:45.542895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4ce2cd286b2c41b89ab361aa3bc2dd18)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:45.555479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:45.678866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:45.686800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4ce2cd286b2c41b89ab361aa3bc2dd18', 'request_id': 'req-7844697a-e325-41f1-b59b-6bdec3324ab3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:45.703459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4ce2cd286b2c41b89ab361aa3bc2dd18', 'request_id': 'req-7844697a-e325-41f1-b59b-6bdec3324ab3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:46.245051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4ce2cd286b2c41b89ab361aa3bc2dd18', 'request_id': 'req-7844697a-e325-41f1-b59b-6bdec3324ab3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:46.257017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] The token cache is being invalidate because project 4ce2cd286b2c41b89ab361aa3bc2dd18 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:46.257310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:46.257949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-7844697a-e325-41f1-b59b-6bdec3324ab3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:46.259550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1016/2052] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:45 2022] DELETE /identity/v3/projects/4ce2cd286b2c41b89ab361aa3bc2dd18 => generated 0 bytes in 756 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:46.265442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2e590-d725-485c-a228-8d48749b2871 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.265954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2e590-d725-485c-a228-8d48749b2871 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.266066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48b2e590-d725-485c-a228-8d48749b2871 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.283428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-48b2e590-d725-485c-a228-8d48749b2871 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:46.284634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-48b2e590-d725-485c-a228-8d48749b2871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:46.363122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1037/2053] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:46.373229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c80bf9-3c91-49eb-903e-43a0eb0ff862 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.373343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c80bf9-3c91-49eb-903e-43a0eb0ff862 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.373654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c80bf9-3c91-49eb-903e-43a0eb0ff862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.392152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-68c80bf9-3c91-49eb-903e-43a0eb0ff862 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:46.393439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-68c80bf9-3c91-49eb-903e-43a0eb0ff862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:46.402607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1017/2054] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:46.407136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2e590-d725-485c-a228-8d48749b2871 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:46.432083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48b2e590-d725-485c-a228-8d48749b2871 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:46.439714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:46.440442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.440729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.441006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.442216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:46.442531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:46.444611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:46.461238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1038/2055] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:46.462407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c80bf9-3c91-49eb-903e-43a0eb0ff862 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:46.473098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c80bf9-3c91-49eb-903e-43a0eb0ff862 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:46.478921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:46.480214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.480585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.480949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.481667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:46.485051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:46.587643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1018/2056] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:46 2022] POST /identity/v3/projects => generated 385 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:46.592642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:46.603929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbd36eb-b797-4986-96ae-62c84bf332b2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:46.676373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:46.677857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.677857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.677857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.678314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:46.680462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:46.806299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:46.822585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1039/2057] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:46 2022] POST /identity/v3/users => generated 293 bytes in 231 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:46.827528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:46.840200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e630f58e-0263-4468-b339-26a1cf5804ca admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:46.845571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:46.846599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.846599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.846848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.847550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:46.849182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:46.860122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1019/2058] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:46.864172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:46.874234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3438457a-9105-497f-9ba5-123631b8a84b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:46.878858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb076d80-b338-4261-b8f0-f777f5d39341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:46.879518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb076d80-b338-4261-b8f0-f777f5d39341 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.879802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb076d80-b338-4261-b8f0-f777f5d39341 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.880095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb076d80-b338-4261-b8f0-f777f5d39341 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.880796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb076d80-b338-4261-b8f0-f777f5d39341 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:46.882387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb076d80-b338-4261-b8f0-f777f5d39341 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:46.893784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1040/2059] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:46 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:46.897987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:46.908485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fd3849-c8a7-4c87-b24a-c3493f938b48 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:46.913862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:46.914738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:46.915055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:46.915328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] PATH_INFO: `/v3/projects/5c461387052a4fc582d5117745396fe5/users/e416414521dd43f682810ec68deae013/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:46.931860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c461387052a4fc582d5117745396fe5, user_id=e416414521dd43f682810ec68deae013, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:46.934655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:47.068446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1020/2060] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:46 2022] PUT /identity/v3/projects/5c461387052a4fc582d5117745396fe5/users/e416414521dd43f682810ec68deae013/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:47.081052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345520ce-abb2-4179-b853-1b5e0c147369 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:47.081340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345520ce-abb2-4179-b853-1b5e0c147369 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:47.081646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345520ce-abb2-4179-b853-1b5e0c147369 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:47.110844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-345520ce-abb2-4179-b853-1b5e0c147369 None None] MFA Rules not processed for user `e416414521dd43f682810ec68deae013`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:47.112111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-345520ce-abb2-4179-b853-1b5e0c147369 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:47.178146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1041/2061] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:47.202574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:47.205014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:47.276082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfffd286-996a-472c-be32-b0b2fd45b359 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:47.281306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8530f5b-0a42-4558-ac11-2e52dcd56cda service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:47.282230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8530f5b-0a42-4558-ac11-2e52dcd56cda service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:47.282525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8530f5b-0a42-4558-ac11-2e52dcd56cda service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:47.282831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8530f5b-0a42-4558-ac11-2e52dcd56cda service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:47.293128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8530f5b-0a42-4558-ac11-2e52dcd56cda service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:47.295123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8530f5b-0a42-4558-ac11-2e52dcd56cda service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:47.305198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1021/2062] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:47.332080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345520ce-abb2-4179-b853-1b5e0c147369 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:47.420757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-345520ce-abb2-4179-b853-1b5e0c147369 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:47.428018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:47.428994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:47.429310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:47.429556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] PATH_INFO: `/v3/users/e416414521dd43f682810ec68deae013` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:47.431342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e416414521dd43f682810ec68deae013)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:47.434282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:47.624007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e416414521dd43f682810ec68deae013'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:47.719302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e416414521dd43f682810ec68deae013'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:47.743196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e416414521dd43f682810ec68deae013'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:47.743947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e416414521dd43f682810ec68deae013'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:47.746275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e416414521dd43f682810ec68deae013'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:47.761449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:47.790383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1042/2063] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:47 2022] DELETE /identity/v3/users/e416414521dd43f682810ec68deae013 => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:47.792016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:47.792429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:47.792720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:47.814933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:47.816373 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:47.910398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1022/2064] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:47.935394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:47.938046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:48.053066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c0af1-5415-4023-9692-0991c880db45 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:48.065504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bfbf30-9305-4a1b-a21c-573880ab5aa4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:48.066553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bfbf30-9305-4a1b-a21c-573880ab5aa4 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:48.067023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bfbf30-9305-4a1b-a21c-573880ab5aa4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:48.067416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bfbf30-9305-4a1b-a21c-573880ab5aa4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:48.089888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bfbf30-9305-4a1b-a21c-573880ab5aa4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:48.091708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bfbf30-9305-4a1b-a21c-573880ab5aa4 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:48.109707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1043/2065] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:48.951369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:48.953507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:48.966292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6fdffa4-ab62-40b0-bf28-846ee43edfa4 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:48.971999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:48.973160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:48.973650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:48.974546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] PATH_INFO: `/v3/projects/5c461387052a4fc582d5117745396fe5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:48.976056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c461387052a4fc582d5117745396fe5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:48.978227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:48.982611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:49.152079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c461387052a4fc582d5117745396fe5', 'request_id': 'req-085cd0a7-b7ee-448c-984e-3667872937ef'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:49.158557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c461387052a4fc582d5117745396fe5', 'request_id': 'req-085cd0a7-b7ee-448c-984e-3667872937ef'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:49.263465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c461387052a4fc582d5117745396fe5', 'request_id': 'req-085cd0a7-b7ee-448c-984e-3667872937ef'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:49.280964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] The token cache is being invalidate because project 5c461387052a4fc582d5117745396fe5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:49.281315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:49.282237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-085cd0a7-b7ee-448c-984e-3667872937ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:49.289901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1023/2066] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:48 2022] DELETE /identity/v3/projects/5c461387052a4fc582d5117745396fe5 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:49.291654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcace82f-5ed9-46e8-ad63-7457e1f7912b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:49.292785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcace82f-5ed9-46e8-ad63-7457e1f7912b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:49.292785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcace82f-5ed9-46e8-ad63-7457e1f7912b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:49.317333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bcace82f-5ed9-46e8-ad63-7457e1f7912b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:49.318975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bcace82f-5ed9-46e8-ad63-7457e1f7912b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:49.411934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1044/2067] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:49.420640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c58ede-6dc4-4f9b-8b91-cb17789a26be None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:49.420930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c58ede-6dc4-4f9b-8b91-cb17789a26be None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:49.421211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c58ede-6dc4-4f9b-8b91-cb17789a26be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:49.447923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-05c58ede-6dc4-4f9b-8b91-cb17789a26be None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:49.451857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-05c58ede-6dc4-4f9b-8b91-cb17789a26be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:49.475993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1024/2068] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:49.481492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcace82f-5ed9-46e8-ad63-7457e1f7912b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:49.491650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcace82f-5ed9-46e8-ad63-7457e1f7912b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:49.496817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:49.497518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:49.497785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:49.498066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:49.506072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:49.506449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:49.508147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:49.539013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1045/2069] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:49.539670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c58ede-6dc4-4f9b-8b91-cb17789a26be None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:49.556115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c58ede-6dc4-4f9b-8b91-cb17789a26be None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:49.563822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:49.564671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:49.564960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:49.565258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:49.565967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:49.567912 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:49.668024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1025/2070] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:49 2022] POST /identity/v3/projects => generated 373 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:49.669303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:49.681571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d060d-c63b-4547-a944-9651b9a33bcb admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:49.786213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:49.786213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:49.786213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:49.786213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:49.786885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:49.786885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:49.975990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:49.985692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1046/2071] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:49 2022] POST /identity/v3/users => generated 287 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:49.993187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-736e91c6-0abc-4cfc-a3ab-c2fd761e3703 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:50.004763 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:50.005897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:50.006356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:50.006845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:50.007892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:50.010056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:50.030282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1026/2072] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:49 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:19:50.034029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:50.046633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d546ae2-63d7-43eb-8732-74028201ff92 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:50.053303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee3dd6c4-44fa-41b4-98e8-379c92450228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:50.053991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3dd6c4-44fa-41b4-98e8-379c92450228 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:50.054278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3dd6c4-44fa-41b4-98e8-379c92450228 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:50.054558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee3dd6c4-44fa-41b4-98e8-379c92450228 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:50.055344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3dd6c4-44fa-41b4-98e8-379c92450228 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:50.056968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee3dd6c4-44fa-41b4-98e8-379c92450228 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:50.069294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1047/2073] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:50 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:50.073543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:50.085850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e80cf625-1f2b-4dcf-aca0-1f67e02451e8 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:50.094965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:50.095919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:50.096732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:50.098624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] PATH_INFO: `/v3/projects/dfa18dfa445442d4be02d187ed24325f/users/7489ed79115044388225314714876f24/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:50.124854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dfa18dfa445442d4be02d187ed24325f, user_id=7489ed79115044388225314714876f24, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:50.127998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:50.229023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1027/2074] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:50 2022] PUT /identity/v3/projects/dfa18dfa445442d4be02d187ed24325f/users/7489ed79115044388225314714876f24/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:50.235210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:50.235732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:50.236193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:50.258605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:50.271242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:50.278959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:50.386747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1048/2075] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:50.408429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:50.411750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:50.533038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3092a7f-aa36-4c6e-98df-28ceda63b304 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:50.538323 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff36e22-5d97-40a8-89f0-50ea447d5a5b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:50.539271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff36e22-5d97-40a8-89f0-50ea447d5a5b service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:50.539864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff36e22-5d97-40a8-89f0-50ea447d5a5b service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:50.540350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff36e22-5d97-40a8-89f0-50ea447d5a5b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:50.563987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff36e22-5d97-40a8-89f0-50ea447d5a5b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:50.563987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff36e22-5d97-40a8-89f0-50ea447d5a5b service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:50.579985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1028/2076] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:51.801136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:51.802862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:51.818666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65839d6-a7a5-4c45-a683-cdde44e8e182 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:51.835950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:51.836779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:51.837746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:51.838356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] PATH_INFO: `/v3/users/55b20ca62ddd422ab1722a2bad96e8a9` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:51.845885 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55b20ca62ddd422ab1722a2bad96e8a9)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:51.848297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:52.126238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55b20ca62ddd422ab1722a2bad96e8a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:52.227012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55b20ca62ddd422ab1722a2bad96e8a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:52.244494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55b20ca62ddd422ab1722a2bad96e8a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:52.246076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55b20ca62ddd422ab1722a2bad96e8a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:52.247734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55b20ca62ddd422ab1722a2bad96e8a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:52.284111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1049/2077] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:51 2022] DELETE /identity/v3/users/55b20ca62ddd422ab1722a2bad96e8a9 => generated 0 bytes in 483 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:52.288327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:52.288843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:52.289328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:52.310207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:52.311482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:52.342052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:52.378371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1029/2078] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:52.389162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:52.391948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:52.462966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3369fcd1-d13c-482e-9b1a-7c08c7cfeb59 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:52.467767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66f534b3-ad4d-4bf6-9285-3faa03ea8103 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:52.468434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f534b3-ad4d-4bf6-9285-3faa03ea8103 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:52.468719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f534b3-ad4d-4bf6-9285-3faa03ea8103 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:52.468998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66f534b3-ad4d-4bf6-9285-3faa03ea8103 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:52.498614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f534b3-ad4d-4bf6-9285-3faa03ea8103 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:52.504262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66f534b3-ad4d-4bf6-9285-3faa03ea8103 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:52.516697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1050/2079] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:53.131203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:53.131203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:53.132717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:53.133174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:53.133275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:53.166548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e338e4a5-f9ea-404f-8f60-d7e6606bbb9b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:53.181679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:53.183505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:53.183811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:53.184106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] PATH_INFO: `/v3/projects/e28c4726590a468490fac2564379e210` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:53.185877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] RBAC: Authorizing `identity:delete_project(project_id=e28c4726590a468490fac2564379e210)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:53.187951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:53.203464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] MFA Rules not processed for user `15e708082e294dd9a0834ef21ea46fd9`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:53.206971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:53.222802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:53.342878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1051/2080] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:53.344610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e28c4726590a468490fac2564379e210', 'request_id': 'req-b8fe175d-60f1-4141-adf9-16fa23b6d4da'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:53.355082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:53.357195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:53.357337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e28c4726590a468490fac2564379e210', 'request_id': 'req-b8fe175d-60f1-4141-adf9-16fa23b6d4da'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:53.422338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e28c4726590a468490fac2564379e210', 'request_id': 'req-b8fe175d-60f1-4141-adf9-16fa23b6d4da'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:53.441492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] The token cache is being invalidate because project e28c4726590a468490fac2564379e210 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:19:53.441785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:53.444086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b8fe175d-60f1-4141-adf9-16fa23b6d4da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:53.448243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1030/2081] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:19:53 2022] DELETE /identity/v3/projects/e28c4726590a468490fac2564379e210 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:53.453974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bd3a1-910f-4664-9f4a-2cc22afc0127 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:53.454386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bd3a1-910f-4664-9f4a-2cc22afc0127 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:53.456224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0bd3a1-910f-4664-9f4a-2cc22afc0127 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:53.498188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66b2208-3a78-4da7-8ebd-868e82e7ee3a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:53.501450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4d0bd3a1-910f-4664-9f4a-2cc22afc0127 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:53.502729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4d0bd3a1-910f-4664-9f4a-2cc22afc0127 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:53.619221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1031/2082] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:53.622965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb747221-22eb-4a15-a650-ad891d51345d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:53.623645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb747221-22eb-4a15-a650-ad891d51345d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:53.623897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb747221-22eb-4a15-a650-ad891d51345d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:53.636132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:53.638312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:53.641343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:53.641728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:53.650314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:53.651678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-fb747221-22eb-4a15-a650-ad891d51345d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:53.653506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fb747221-22eb-4a15-a650-ad891d51345d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:53.684992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1032/2083] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:53.689511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb747221-22eb-4a15-a650-ad891d51345d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:53.703074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb747221-22eb-4a15-a650-ad891d51345d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:53.709679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:53.715829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:53.716527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:53.716807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:53.718206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:19:53.719982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:53.727438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:53.782716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1033/2084] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:19:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:19:53.787770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:53.790426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:53.797060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:53.825354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5504491-f2b0-497d-bea0-007b064a24a6 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:53.835531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1052/2085] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:53 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:53.876515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:53.880512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:53.880911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:53.881605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:53.883166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:53.888230 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:53.998538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1034/2086] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:19:53 2022] POST /identity/v3/projects => generated 377 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:19:54.002428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:54.014228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea40011-6881-467b-b517-32381b2ea34e service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:54.091983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:54.092961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:54.093411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:54.093841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:54.095960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:54.098134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:54.210858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1053/2087] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:19:54 2022] POST /identity/v3/users => generated 289 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:19:54.212694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:54.228328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:54.230729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35a476f-6ddd-4c61-8c8c-0861356c9379 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:54.242742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:54.243553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:54.243929 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:54.244298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:54.245183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:54.247143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:54.265551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1035/2088] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:54.270880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:54.283327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd924737-b029-49bb-958c-a975e0b1450f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:54.288987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33ec437a-e736-49ac-9fc6-02938b292057 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:54.298172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec437a-e736-49ac-9fc6-02938b292057 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:54.298172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec437a-e736-49ac-9fc6-02938b292057 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:54.298172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33ec437a-e736-49ac-9fc6-02938b292057 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:54.298172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec437a-e736-49ac-9fc6-02938b292057 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:54.298172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33ec437a-e736-49ac-9fc6-02938b292057 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:54.312992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1054/2089] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:19:54 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:19:54.315330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:54.320582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:54.326753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd171ca7-79e5-4f77-854d-5870c4449cf4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:54.347048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:54.347958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:54.348236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:54.348509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] PATH_INFO: `/v3/projects/a7383992d769483fae0ef92d4b5ec489/users/ef6ef54116844f419dc1e50f5c1f37a6/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:54.400421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=a7383992d769483fae0ef92d4b5ec489, user_id=ef6ef54116844f419dc1e50f5c1f37a6, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:54.404356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:54.495594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1036/2090] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:19:54 2022] PUT /identity/v3/projects/a7383992d769483fae0ef92d4b5ec489/users/ef6ef54116844f419dc1e50f5c1f37a6/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:19:54.508556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b7f2de-23a4-4e34-bd5d-d3003481607a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:54.509171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b7f2de-23a4-4e34-bd5d-d3003481607a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:54.509627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b7f2de-23a4-4e34-bd5d-d3003481607a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:54.552625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-54b7f2de-23a4-4e34-bd5d-d3003481607a None None] MFA Rules not processed for user `ef6ef54116844f419dc1e50f5c1f37a6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:54.554069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-54b7f2de-23a4-4e34-bd5d-d3003481607a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:54.700826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1055/2091] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:54 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:54.720227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:54.723628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:54.791857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b56377e-edf4-400f-a604-967e1e76bb4e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:54.797862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:54.798731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:54.799148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:54.799497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:54.809666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:54.811662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:54.837303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1037/2092] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:54 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:57.836090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac31a26-4070-45da-b987-ac0b34d7e3f7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:57.836644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac31a26-4070-45da-b987-ac0b34d7e3f7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:57.837236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac31a26-4070-45da-b987-ac0b34d7e3f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:57.860749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4ac31a26-4070-45da-b987-ac0b34d7e3f7 None None] MFA Rules not processed for user `15e708082e294dd9a0834ef21ea46fd9`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:57.862329 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4ac31a26-4070-45da-b987-ac0b34d7e3f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:57.925911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:57.941752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1056/2093] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:57 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:19:58.601666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:58.603584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:58.752517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8613c25d-032f-402e-9c5e-b59394e174ad service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:58.762746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:58.767089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:58.767089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:58.767089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] PATH_INFO: `/v3/users/c546618fc28d40e9aa015a4ba8482351` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:58.767089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c546618fc28d40e9aa015a4ba8482351)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:58.768910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:58.973672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c546618fc28d40e9aa015a4ba8482351'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:59.046510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c546618fc28d40e9aa015a4ba8482351'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:59.047383 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c546618fc28d40e9aa015a4ba8482351'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:59.068012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c546618fc28d40e9aa015a4ba8482351'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:59.069179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c546618fc28d40e9aa015a4ba8482351'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:19:59.086799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:19:59.091857 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1038/2094] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:19:58 2022] DELETE /identity/v3/users/c546618fc28d40e9aa015a4ba8482351 => generated 0 bytes in 491 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:19:59.096361 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8e8ea1-9156-42ae-a33c-5c048cbea676 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:59.096868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8e8ea1-9156-42ae-a33c-5c048cbea676 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:59.097067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8e8ea1-9156-42ae-a33c-5c048cbea676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:59.122551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ca8e8ea1-9156-42ae-a33c-5c048cbea676 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:59.123874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ca8e8ea1-9156-42ae-a33c-5c048cbea676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:59.197606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1057/2095] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:59 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:19:59.208826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:19:59.210563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:19:59.314731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0f3374f-fef3-48c5-9464-374fe41a3d83 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:19:59.319083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:19:59.319882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:59.320285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:59.320548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:59.340720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:19:59.343398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:19:59.365131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1039/2096] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:19:59 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:19:59.893069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba235931-08e4-47bd-882e-80366813381d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:19:59.893358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba235931-08e4-47bd-882e-80366813381d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:19:59.893635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba235931-08e4-47bd-882e-80366813381d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:19:59.927262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ba235931-08e4-47bd-882e-80366813381d None None] MFA Rules not processed for user `7489ed79115044388225314714876f24`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:19:59.928496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ba235931-08e4-47bd-882e-80366813381d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:00.013636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1058/2097] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:19:59 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:00.017818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.019408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:00.034054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfa3524-6898-4ffc-a653-ade08bb303a1 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.039470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.040376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.040668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.041582 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.041943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.043930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.145091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1040/2098] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:00 2022] POST /identity/v3/projects => generated 371 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:00.149176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba235931-08e4-47bd-882e-80366813381d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.165014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba235931-08e4-47bd-882e-80366813381d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.226285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.227017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.227303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.227777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.228496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.230758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.258093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.259948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:00.275040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5761b701-9b33-4b55-a502-48fba9e64ce1 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.281092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.282378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.284515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.284515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] PATH_INFO: `/v3/projects/636e780d917d4b96bbafd985d415997e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.285021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=636e780d917d4b96bbafd985d415997e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.287962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.365736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1059/2099] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:00 2022] POST /identity/v3/users => generated 292 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:00.369599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.382926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb39e345-101c-4df9-aa71-631c0fd3595c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.388816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.389581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.389937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.390289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.391140 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.392830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.406338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1060/2100] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:00 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:20:00.410665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.411966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '636e780d917d4b96bbafd985d415997e', 'request_id': 'req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.421133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '636e780d917d4b96bbafd985d415997e', 'request_id': 'req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.422704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097e9348-b235-4304-b27e-9c0eb54e1b84 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.428832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:00.485801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.486708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.487077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.487364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] PATH_INFO: `/v3/projects/8d43e94a04ab44dda0f87babe90a02be/users/1aec8c73c2dd4021b4740fbad1f51adc/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.518940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d43e94a04ab44dda0f87babe90a02be, user_id=1aec8c73c2dd4021b4740fbad1f51adc, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.518940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.536349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '636e780d917d4b96bbafd985d415997e', 'request_id': 'req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.552785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] The token cache is being invalidate because project 636e780d917d4b96bbafd985d415997e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:00.553107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.553791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.555500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1041/2101] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:00 2022] DELETE /identity/v3/projects/636e780d917d4b96bbafd985d415997e => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:00.561236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.562694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:00.577437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-758c0cc8-9a6b-4fc5-8ee5-339a0e9ffef9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.583280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.584298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.584702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.585077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] PATH_INFO: `/v3/users/af701bcd5f0c471384344cd671e3b2fe` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.586429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=af701bcd5f0c471384344cd671e3b2fe)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.588732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.608268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1061/2102] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:00 2022] PUT /identity/v3/projects/8d43e94a04ab44dda0f87babe90a02be/users/1aec8c73c2dd4021b4740fbad1f51adc/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:00.612261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.616268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:00.690117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f545c4f1-e051-47aa-b1b8-a5a6733fa1e2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.694945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.695642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.695921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.696193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.696968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.698626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.708966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1062/2103] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:00 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:20:00.712797 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:00.723620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ced32f6-313c-44bb-9cc1-6fda4f113db6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:00.730397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:00.730722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:00.731124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:00.731478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] PATH_INFO: `/v3/projects/8d43e94a04ab44dda0f87babe90a02be/users/1aec8c73c2dd4021b4740fbad1f51adc/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:00.733945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d43e94a04ab44dda0f87babe90a02be, user_id=1aec8c73c2dd4021b4740fbad1f51adc, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:00.736712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:00.747892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1063/2104] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:00 2022] PUT /identity/v3/projects/8d43e94a04ab44dda0f87babe90a02be/users/1aec8c73c2dd4021b4740fbad1f51adc/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:00.764886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af701bcd5f0c471384344cd671e3b2fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.868169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af701bcd5f0c471384344cd671e3b2fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.868923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af701bcd5f0c471384344cd671e3b2fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.874472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=82,keystone:INSERT=3,keystone:DELETE=15 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:00.877484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af701bcd5f0c471384344cd671e3b2fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.878237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-221ef362-39ae-4777-89ef-0dcfb759ea6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af701bcd5f0c471384344cd671e3b2fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:00.896101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1042/2105] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:00 2022] DELETE /identity/v3/users/af701bcd5f0c471384344cd671e3b2fe => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:01.494179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:01.495169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:01.595469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0da11d3-a3ee-4a92-8559-b3fad63febd8 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:01.633905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:01.745680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:01.746945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:01.747524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:01.747818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] PATH_INFO: `/v3/projects/34fe7c1656d844e9b920116305e053b6` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:01.750910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] RBAC: Authorizing `identity:delete_project(project_id=34fe7c1656d844e9b920116305e053b6)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:01.754834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:01.988507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34fe7c1656d844e9b920116305e053b6', 'request_id': 'req-76f1274d-fc72-43aa-a864-826e7befbe96'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:02.080977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34fe7c1656d844e9b920116305e053b6', 'request_id': 'req-76f1274d-fc72-43aa-a864-826e7befbe96'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:02.088576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34fe7c1656d844e9b920116305e053b6', 'request_id': 'req-76f1274d-fc72-43aa-a864-826e7befbe96'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:02.124795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] The token cache is being invalidate because project 34fe7c1656d844e9b920116305e053b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:02.125670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:02.131901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76f1274d-fc72-43aa-a864-826e7befbe96 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:02.132676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1064/2106] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:01 2022] DELETE /identity/v3/projects/34fe7c1656d844e9b920116305e053b6 => generated 0 bytes in 640 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:02.137272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2e0f5-853d-464e-b803-33d1b94763f3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.137759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2e0f5-853d-464e-b803-33d1b94763f3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.137916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0b2e0f5-853d-464e-b803-33d1b94763f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.171597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b0b2e0f5-853d-464e-b803-33d1b94763f3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:02.172991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b0b2e0f5-853d-464e-b803-33d1b94763f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:02.265267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1043/2107] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:02.271096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97d6977-b0b2-4722-a53e-357e7abc4c70 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.271391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97d6977-b0b2-4722-a53e-357e7abc4c70 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.271756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a97d6977-b0b2-4722-a53e-357e7abc4c70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.291040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a97d6977-b0b2-4722-a53e-357e7abc4c70 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:02.292608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a97d6977-b0b2-4722-a53e-357e7abc4c70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:02.301652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1065/2108] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:02.306148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2e0f5-853d-464e-b803-33d1b94763f3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:02.317033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0b2e0f5-853d-464e-b803-33d1b94763f3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:02.321656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:02.322314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.322640 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.323616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.324356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:02.324670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:02.326292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:02.338634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1044/2109] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:20:02.342996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97d6977-b0b2-4722-a53e-357e7abc4c70 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:02.352405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a97d6977-b0b2-4722-a53e-357e7abc4c70 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:02.357650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:02.358253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.358538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.358856 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.359587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:02.361504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:02.493516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1066/2110] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:02 2022] POST /identity/v3/projects => generated 387 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:02.494725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:02.507591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db7f500a-479d-4679-9dd1-bb5c3c2ea282 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:02.632432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:02.633883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.633883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.634060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.634719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:02.636827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:02.743399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1045/2111] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:02 2022] POST /identity/v3/users => generated 294 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:02.748073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:02.761447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac18f5d0-8125-4af5-b2bb-526546f12a31 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:02.774724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:02.775755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.776275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.776721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.779063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:02.781179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:02.798320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1067/2112] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:02 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:02.799272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:02.812841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a57eb3-830f-48e4-ab32-6444c15ece90 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:02.821445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd35deb-623a-45d7-9662-c0767e0c5aba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:02.822140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd35deb-623a-45d7-9662-c0767e0c5aba admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.822423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd35deb-623a-45d7-9662-c0767e0c5aba admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.822716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd35deb-623a-45d7-9662-c0767e0c5aba admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.824271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd35deb-623a-45d7-9662-c0767e0c5aba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:02.826074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fd35deb-623a-45d7-9662-c0767e0c5aba admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:02.844964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1046/2113] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:02 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:02.846453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:02.862139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0582ea2-db17-4089-9d22-4a76368b83e6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:02.867575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:02.868515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.868784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.869065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] PATH_INFO: `/v3/projects/afd22ff2769a4fa2be3435cb68c748f5/users/20544526f9ad4bb2ad968f35db7d17c7/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:02.894712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=afd22ff2769a4fa2be3435cb68c748f5, user_id=20544526f9ad4bb2ad968f35db7d17c7, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:02.897595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:02.982246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1068/2114] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:02 2022] PUT /identity/v3/projects/afd22ff2769a4fa2be3435cb68c748f5/users/20544526f9ad4bb2ad968f35db7d17c7/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:02.993624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f15b61-5e9e-4cd9-a75d-a128717e82a4 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:02.994017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f15b61-5e9e-4cd9-a75d-a128717e82a4 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:02.994302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54f15b61-5e9e-4cd9-a75d-a128717e82a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:03.029486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-54f15b61-5e9e-4cd9-a75d-a128717e82a4 None None] MFA Rules not processed for user `20544526f9ad4bb2ad968f35db7d17c7`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:03.030882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-54f15b61-5e9e-4cd9-a75d-a128717e82a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:03.089133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:03.144518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1047/2115] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:03.150376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:03.252745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:03.279510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7665e8fb-fe5c-42cd-b460-c62a3da1dda2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:03.291207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:03.292114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:03.292403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:03.296203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:03.296203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:03.299853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:03.397405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1069/2116] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:03 2022] POST /identity/v3/projects => generated 383 bytes in 247 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:03.403231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f15b61-5e9e-4cd9-a75d-a128717e82a4 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:03.422044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54f15b61-5e9e-4cd9-a75d-a128717e82a4 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:03.548906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:03.549877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:03.550167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:03.550441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:03.553397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:03.556107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:03.636914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1048/2117] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:03 2022] POST /identity/v3/users => generated 298 bytes in 221 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:03.637601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:03.641752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7026c2-116b-44f3-b2c4-5614f4abffb2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:03.647978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:03.649126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:03.649126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:03.649334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:03.650082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:03.652963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:03.666649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1070/2118] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:03 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:20:03.671949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:03.686047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e9c574-0399-4045-9a7f-e16c5d1864b0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:03.691910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:03.697275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:03.697275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:03.697275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] PATH_INFO: `/v3/projects/805a2884a5774256912a890fd1acb1ed/users/fdfdf3bc507342e596555d880dd1aa04/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:03.727262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] RBAC: Authorizing `identity:create_grant(project_id=805a2884a5774256912a890fd1acb1ed, user_id=fdfdf3bc507342e596555d880dd1aa04, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:03.735728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:03.808881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1049/2119] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:03 2022] PUT /identity/v3/projects/805a2884a5774256912a890fd1acb1ed/users/fdfdf3bc507342e596555d880dd1aa04/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:03.812961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:03.823817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6c44f9e-ba9c-49a6-b161-6d42574fee0f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:03.923129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525b130c-2edb-4511-b673-943e2c658a21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:03.923981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525b130c-2edb-4511-b673-943e2c658a21 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:03.924368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525b130c-2edb-4511-b673-943e2c658a21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:03.924673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525b130c-2edb-4511-b673-943e2c658a21 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:03.925556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525b130c-2edb-4511-b673-943e2c658a21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:03.927553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525b130c-2edb-4511-b673-943e2c658a21 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:03.961599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1071/2120] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:03 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:03.966032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:03.989343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893aba2b-8b20-415e-9690-f316364bd372 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:04.004562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:04.005484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:04.005769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:04.006385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] PATH_INFO: `/v3/projects/805a2884a5774256912a890fd1acb1ed/users/fdfdf3bc507342e596555d880dd1aa04/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:04.027105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=805a2884a5774256912a890fd1acb1ed, user_id=fdfdf3bc507342e596555d880dd1aa04, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:04.027105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:04.082116 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1050/2121] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:03 2022] PUT /identity/v3/projects/805a2884a5774256912a890fd1acb1ed/users/fdfdf3bc507342e596555d880dd1aa04/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:04.097199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5251ee6-cfc4-4856-8e91-44527ee33d88 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:04.097199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5251ee6-cfc4-4856-8e91-44527ee33d88 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:04.097199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5251ee6-cfc4-4856-8e91-44527ee33d88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:04.161235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c5251ee6-cfc4-4856-8e91-44527ee33d88 None None] MFA Rules not processed for user `fdfdf3bc507342e596555d880dd1aa04`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:04.162653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c5251ee6-cfc4-4856-8e91-44527ee33d88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:04.288911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1072/2122] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:04 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:04.301672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:04.304823 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:04.394720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4165e83-3895-4ba2-87f5-55c85a98f1ae admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:04.402514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0eca15b-cc1c-462f-bec3-635134a9414a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:04.403551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eca15b-cc1c-462f-bec3-635134a9414a service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:04.404110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eca15b-cc1c-462f-bec3-635134a9414a service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:04.408100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0eca15b-cc1c-462f-bec3-635134a9414a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:04.414485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eca15b-cc1c-462f-bec3-635134a9414a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:04.415981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0eca15b-cc1c-462f-bec3-635134a9414a service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:04.428276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1051/2123] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:04 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:04.512981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5251ee6-cfc4-4856-8e91-44527ee33d88 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:04.575993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:04.595653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5251ee6-cfc4-4856-8e91-44527ee33d88 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:04.600333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:04.601322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:04.601772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:04.602123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] PATH_INFO: `/v3/users/20544526f9ad4bb2ad968f35db7d17c7` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:04.606968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] RBAC: Authorizing `identity:delete_user(user_id=20544526f9ad4bb2ad968f35db7d17c7)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:04.609213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:04.827215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '20544526f9ad4bb2ad968f35db7d17c7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:04.908351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '20544526f9ad4bb2ad968f35db7d17c7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:04.917179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '20544526f9ad4bb2ad968f35db7d17c7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:04.918240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '20544526f9ad4bb2ad968f35db7d17c7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:04.918947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '20544526f9ad4bb2ad968f35db7d17c7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:04.940755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1073/2124] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:04 2022] DELETE /identity/v3/users/20544526f9ad4bb2ad968f35db7d17c7 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:04.945111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:04.946874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:04.946874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:04.964928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:04.966151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:04.987593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:05.032676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1052/2125] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:20:05.043527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:05.045636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:05.114789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67137ddb-bd17-431a-8974-292935ecc502 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:05.118595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:05.119375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:05.119752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:05.120632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:05.133837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:05.139350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:05.153741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1074/2126] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:05.812627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:05.814242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:05.828837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcad5f33-1f0c-4067-b9e1-8c7273bd8c74 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:05.835277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:05.836770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:05.837424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:05.837949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] PATH_INFO: `/v3/projects/afd22ff2769a4fa2be3435cb68c748f5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:05.840381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] RBAC: Authorizing `identity:delete_project(project_id=afd22ff2769a4fa2be3435cb68c748f5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:05.842660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:06.010800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'afd22ff2769a4fa2be3435cb68c748f5', 'request_id': 'req-6b84c96f-da67-42e3-a242-7fd83091f644'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.031046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'afd22ff2769a4fa2be3435cb68c748f5', 'request_id': 'req-6b84c96f-da67-42e3-a242-7fd83091f644'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.130477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'afd22ff2769a4fa2be3435cb68c748f5', 'request_id': 'req-6b84c96f-da67-42e3-a242-7fd83091f644'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.145483 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] The token cache is being invalidate because project afd22ff2769a4fa2be3435cb68c748f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:06.145782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.146504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.148127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1053/2127] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:05 2022] DELETE /identity/v3/projects/afd22ff2769a4fa2be3435cb68c748f5 => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:06.153119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:06.163396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13aab752-1f7b-4f68-9ab0-ef99509efe11 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:06.233556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:06.234622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:06.235069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:06.235477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] PATH_INFO: `/v3/users/fdfdf3bc507342e596555d880dd1aa04` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:06.250050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] RBAC: Authorizing `identity:delete_user(user_id=fdfdf3bc507342e596555d880dd1aa04)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:06.255650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:06.269917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:06.478137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fdfdf3bc507342e596555d880dd1aa04'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.624464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fdfdf3bc507342e596555d880dd1aa04'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.639334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fdfdf3bc507342e596555d880dd1aa04'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.640133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fdfdf3bc507342e596555d880dd1aa04'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.640892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-2c894f1e-03f6-4375-8288-02dfb28c712c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fdfdf3bc507342e596555d880dd1aa04'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:06.663795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1075/2128] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:06 2022] DELETE /identity/v3/users/fdfdf3bc507342e596555d880dd1aa04 => generated 0 bytes in 511 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:07.216054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:07.218741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:07.248921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b84c96f-da67-42e3-a242-7fd83091f644 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:07.332534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:07.333548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:07.333737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:07.333951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] PATH_INFO: `/v3/projects/805a2884a5774256912a890fd1acb1ed` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:07.335285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] RBAC: Authorizing `identity:delete_project(project_id=805a2884a5774256912a890fd1acb1ed)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:07.337217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:07.445326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '805a2884a5774256912a890fd1acb1ed', 'request_id': 'req-b1dcd142-7ac4-40f8-9149-caf5848ee609'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:07.447527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:07.453101 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '805a2884a5774256912a890fd1acb1ed', 'request_id': 'req-b1dcd142-7ac4-40f8-9149-caf5848ee609'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:07.558222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '805a2884a5774256912a890fd1acb1ed', 'request_id': 'req-b1dcd142-7ac4-40f8-9149-caf5848ee609'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:07.592630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] The token cache is being invalidate because project 805a2884a5774256912a890fd1acb1ed was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:07.593282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:07.594179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b1dcd142-7ac4-40f8-9149-caf5848ee609 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:07.597659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1054/2129] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:07 2022] DELETE /identity/v3/projects/805a2884a5774256912a890fd1acb1ed => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:07.602791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1cc458-761a-4f0f-9d27-b5f57720b6f8 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:07.604904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1cc458-761a-4f0f-9d27-b5f57720b6f8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:07.606862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af1cc458-761a-4f0f-9d27-b5f57720b6f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:07.634495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-af1cc458-761a-4f0f-9d27-b5f57720b6f8 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:07.637102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-af1cc458-761a-4f0f-9d27-b5f57720b6f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:07.725000 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1076/2130] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:07.732837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816cc1bc-c6fd-4eb5-a995-3ea3dd8925ed None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:07.732979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816cc1bc-c6fd-4eb5-a995-3ea3dd8925ed None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:07.735175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816cc1bc-c6fd-4eb5-a995-3ea3dd8925ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:07.758117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-816cc1bc-c6fd-4eb5-a995-3ea3dd8925ed None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:07.759455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-816cc1bc-c6fd-4eb5-a995-3ea3dd8925ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:07.773774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1055/2131] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:07.778223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1cc458-761a-4f0f-9d27-b5f57720b6f8 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:07.788713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af1cc458-761a-4f0f-9d27-b5f57720b6f8 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:07.793931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:07.794601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:07.795348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:07.795796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:07.796681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:07.797363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:07.799027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:07.813334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1077/2132] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:20:07.817742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816cc1bc-c6fd-4eb5-a995-3ea3dd8925ed None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:07.829739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816cc1bc-c6fd-4eb5-a995-3ea3dd8925ed None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:07.838374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:07.839423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:07.839716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:07.840734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:07.841484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:07.843641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:07.996011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1056/2133] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:07 2022] POST /identity/v3/projects => generated 401 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:08.001560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:08.046082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db58d33-faff-48a2-8086-ba32cf4daae8 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:08.154936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:08.155634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:08.155919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:08.156196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:08.156917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:08.159495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:08.245747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1078/2134] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:07 2022] POST /identity/v3/users => generated 301 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:08.250198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:08.266497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2ffd2f8-7c27-46c3-8261-fd05a1d262e7 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:08.274682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:08.276915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:08.277381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:08.277666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:08.278425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:08.280341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:08.297190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1057/2135] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:08 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:08.301687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:08.308091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:08.314998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7025ba4a-8c51-4cac-8979-eff9a30a5853 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:08.319757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95f45ea0-1148-447c-8e7e-f25e9de05a02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:08.320491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f45ea0-1148-447c-8e7e-f25e9de05a02 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:08.320778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f45ea0-1148-447c-8e7e-f25e9de05a02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:08.322125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95f45ea0-1148-447c-8e7e-f25e9de05a02 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:08.323015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f45ea0-1148-447c-8e7e-f25e9de05a02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:08.325926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95f45ea0-1148-447c-8e7e-f25e9de05a02 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:08.348489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1079/2136] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:08 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:08.349728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:08.377134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f0f2d-fd00-4914-b6d0-95183c948a98 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:08.382278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:08.383480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:08.383480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:08.383762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] PATH_INFO: `/v3/projects/4d650aec6a634d48872b585b7d67205d/users/a30950a18b1b47b58fcc4aa2f61caeba/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:08.415608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4d650aec6a634d48872b585b7d67205d, user_id=a30950a18b1b47b58fcc4aa2f61caeba, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:08.420913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:08.546407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1058/2137] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:08 2022] PUT /identity/v3/projects/4d650aec6a634d48872b585b7d67205d/users/a30950a18b1b47b58fcc4aa2f61caeba/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:08.548238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24b014-aeb8-4111-9728-4ddc82780221 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:08.548696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24b014-aeb8-4111-9728-4ddc82780221 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:08.548993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c24b014-aeb8-4111-9728-4ddc82780221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:08.593141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1c24b014-aeb8-4111-9728-4ddc82780221 None None] MFA Rules not processed for user `a30950a18b1b47b58fcc4aa2f61caeba`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:08.594457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1c24b014-aeb8-4111-9728-4ddc82780221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:08.679583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1080/2138] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:08 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:08.684029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:08.775171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a189d4c8-0225-4c51-85e8-7e19fb81ae80 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:08.781274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b0aeea1-6bc3-4d6d-9001-33e87ec11dca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:08.781969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0aeea1-6bc3-4d6d-9001-33e87ec11dca admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:08.782270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0aeea1-6bc3-4d6d-9001-33e87ec11dca admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:08.782536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b0aeea1-6bc3-4d6d-9001-33e87ec11dca admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:08.783298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0aeea1-6bc3-4d6d-9001-33e87ec11dca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:08.785269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b0aeea1-6bc3-4d6d-9001-33e87ec11dca admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:08.870858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1059/2139] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:08 2022] POST /identity/v3/projects => generated 401 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:08.874874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c24b014-aeb8-4111-9728-4ddc82780221 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:08.885931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c24b014-aeb8-4111-9728-4ddc82780221 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:08.946778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:08.947509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:08.947831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:08.948124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:08.948837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:08.950778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:09.037136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.037506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.037783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.071243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1081/2140] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:08 2022] POST /identity/v3/users => generated 307 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:09.076835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:09.078153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] MFA Rules not processed for user `1aec8c73c2dd4021b4740fbad1f51adc`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:09.079444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:09.093232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bc33d7-f0c1-4d9f-a7df-41085b6e72c6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:09.100786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:09.101846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.105942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.105942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.105942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:09.107190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:09.134849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1082/2141] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:09 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:09.136835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:09.159532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5be07004-b76e-4117-9534-1e638cfcb2fd admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:09.159791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=94,keystone:DELETE=2,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:09.166028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:09.167736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.168395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.168678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] PATH_INFO: `/v3/projects/3e484fca60594ae8842c9c543d88da20/users/3bd10684088f41c2a2bf30e0e31437a9/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.190350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1060/2142] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:09 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:09.201969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e484fca60594ae8842c9c543d88da20, user_id=3bd10684088f41c2a2bf30e0e31437a9, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:09.204900 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:09.219220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:09.221090 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:09.311180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1083/2143] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:09 2022] PUT /identity/v3/projects/3e484fca60594ae8842c9c543d88da20/users/3bd10684088f41c2a2bf30e0e31437a9/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:09.316172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:09.330500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8136688-eb42-40b6-a2cd-e0be57ce2db2 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:09.365571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:09.396479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974564b5-d974-472e-a1ca-4fb6e899c889 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:09.415357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:09.415735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:09.417176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.418300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.419028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.419150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.419254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.419254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.421241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:09.422396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:09.424007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:09.445910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1084/2144] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:09 2022] GET /identity/v3/roles => generated 1420 bytes in 127 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:09.447143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:09.461063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a97d279-dfc3-4cc0-a772-870f2387e9f3 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:09.469001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23d03c4e-c137-4a00-be3b-ef2837431a85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:09.470030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d03c4e-c137-4a00-be3b-ef2837431a85 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.471210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d03c4e-c137-4a00-be3b-ef2837431a85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.471507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d03c4e-c137-4a00-be3b-ef2837431a85 admin admin] PATH_INFO: `/v3/projects/3e484fca60594ae8842c9c543d88da20/users/3bd10684088f41c2a2bf30e0e31437a9/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.473625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d03c4e-c137-4a00-be3b-ef2837431a85 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e484fca60594ae8842c9c543d88da20, user_id=3bd10684088f41c2a2bf30e0e31437a9, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:09.478068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23d03c4e-c137-4a00-be3b-ef2837431a85 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:09.494527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1085/2145] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:09 2022] PUT /identity/v3/projects/3e484fca60594ae8842c9c543d88da20/users/3bd10684088f41c2a2bf30e0e31437a9/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:09.505012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424e98ab-034e-4173-8dac-483938b1d1d7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.505307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424e98ab-034e-4173-8dac-483938b1d1d7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.505592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424e98ab-034e-4173-8dac-483938b1d1d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.525758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:09.528345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:09.541143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-424e98ab-034e-4173-8dac-483938b1d1d7 None None] MFA Rules not processed for user `3bd10684088f41c2a2bf30e0e31437a9`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:09.543530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-424e98ab-034e-4173-8dac-483938b1d1d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:09.544029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1061/2146] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:09 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:09.633158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1086/2147] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:09 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:20:09.645039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:09.646533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:09.674432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:09.754449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb2a7a0-cf65-4e6a-a8a0-d774c832ed6b service nova] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:09.888597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0608f791-c5c0-43f4-8d78-615a53e49e79 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:09.889415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608f791-c5c0-43f4-8d78-615a53e49e79 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:09.889803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608f791-c5c0-43f4-8d78-615a53e49e79 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:09.891029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608f791-c5c0-43f4-8d78-615a53e49e79 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:09.914087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0608f791-c5c0-43f4-8d78-615a53e49e79 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:09.916259 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0608f791-c5c0-43f4-8d78-615a53e49e79 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:09.943598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1062/2148] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:09 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:10.261992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424e98ab-034e-4173-8dac-483938b1d1d7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:10.275570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424e98ab-034e-4173-8dac-483938b1d1d7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:10.281802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74e2d697-49e8-400b-976b-5818bce758e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:10.282750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2d697-49e8-400b-976b-5818bce758e6 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:10.283364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2d697-49e8-400b-976b-5818bce758e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:10.283917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74e2d697-49e8-400b-976b-5818bce758e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:10.380947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e2d697-49e8-400b-976b-5818bce758e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:10.383096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74e2d697-49e8-400b-976b-5818bce758e6 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:10.401867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1087/2149] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:10 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:13.278130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:13.279666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:13.280107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:13.302588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] MFA Rules not processed for user `7489ed79115044388225314714876f24`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:13.304120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:13.398661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1063/2150] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:13 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:20.289120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b29d5-3720-4800-a58d-82f7b4a9845c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:20.289670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b29d5-3720-4800-a58d-82f7b4a9845c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:20.289781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a06b29d5-3720-4800-a58d-82f7b4a9845c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:20.309397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a06b29d5-3720-4800-a58d-82f7b4a9845c None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:20.310557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a06b29d5-3720-4800-a58d-82f7b4a9845c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:20.367322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:20.388388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1088/2151] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:20:20 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:20.473620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:20.475307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:20.583091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91fe7a2-9ec3-4fc9-a2cd-55b39a57b43c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:20.594160 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20dd999a-3c4d-41e1-83dc-1c8f800b3622 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:20.595161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd999a-3c4d-41e1-83dc-1c8f800b3622 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:20.595709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd999a-3c4d-41e1-83dc-1c8f800b3622 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:20.596115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dd999a-3c4d-41e1-83dc-1c8f800b3622 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:20.620432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20dd999a-3c4d-41e1-83dc-1c8f800b3622 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:20.622177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20dd999a-3c4d-41e1-83dc-1c8f800b3622 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:20.632306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1064/2152] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:20:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:20:20.648903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b29d5-3720-4800-a58d-82f7b4a9845c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:20.821902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a06b29d5-3720-4800-a58d-82f7b4a9845c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:20.843405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:20.844884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:20.844884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:20.844884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] PATH_INFO: `/v3/users/a30950a18b1b47b58fcc4aa2f61caeba` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:20.846748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=a30950a18b1b47b58fcc4aa2f61caeba)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:20.848787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:21.108676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a30950a18b1b47b58fcc4aa2f61caeba'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:21.189971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a30950a18b1b47b58fcc4aa2f61caeba'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:21.207355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a30950a18b1b47b58fcc4aa2f61caeba'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:21.208239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a30950a18b1b47b58fcc4aa2f61caeba'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:21.208896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a30950a18b1b47b58fcc4aa2f61caeba'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:21.259626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1089/2153] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:20 2022] DELETE /identity/v3/users/a30950a18b1b47b58fcc4aa2f61caeba => generated 0 bytes in 605 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:21.259725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:21.259725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:21.259725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:21.279978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:21.281339 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:21.313107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:21.387335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1065/2154] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:20:21.397880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:21.401171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:21.471138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6985a36f-9283-4c5d-b021-f368d4b9c2ea admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:21.475574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:21.476389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:21.477289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:21.477289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:21.488516 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:21.490192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:21.508089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1090/2155] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:22.121713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:22.124328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:22.139497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279fe653-6a3e-4b0e-8abb-c95db490b8e9 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:22.146457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:22.146457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:22.146457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:22.146457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] PATH_INFO: `/v3/projects/4d650aec6a634d48872b585b7d67205d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:22.148548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4d650aec6a634d48872b585b7d67205d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:22.152702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:22.292480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4d650aec6a634d48872b585b7d67205d', 'request_id': 'req-fca2bb78-8096-4964-b9e3-5409f9ab0796'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.298128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4d650aec6a634d48872b585b7d67205d', 'request_id': 'req-fca2bb78-8096-4964-b9e3-5409f9ab0796'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.376382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4d650aec6a634d48872b585b7d67205d', 'request_id': 'req-fca2bb78-8096-4964-b9e3-5409f9ab0796'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.388493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] The token cache is being invalidate because project 4d650aec6a634d48872b585b7d67205d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:22.393152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.393152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.393152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1066/2156] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:22 2022] DELETE /identity/v3/projects/4d650aec6a634d48872b585b7d67205d => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:22.395441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:22.405933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af55d867-10ee-4a7f-b4a3-e99bc6d11a25 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:22.410726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:22.474307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:22.475451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:22.475763 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:22.476046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] PATH_INFO: `/v3/users/3bd10684088f41c2a2bf30e0e31437a9` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:22.477578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bd10684088f41c2a2bf30e0e31437a9)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:22.480063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:22.693647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bd10684088f41c2a2bf30e0e31437a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.760192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bd10684088f41c2a2bf30e0e31437a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.769154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bd10684088f41c2a2bf30e0e31437a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.770013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bd10684088f41c2a2bf30e0e31437a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.770731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-c8616848-da1a-4e02-a00b-a6a99b2a2fc1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bd10684088f41c2a2bf30e0e31437a9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:22.796270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1091/2157] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:22 2022] DELETE /identity/v3/users/3bd10684088f41c2a2bf30e0e31437a9 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:23.443024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:23.443024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:23.481966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca2bb78-8096-4964-b9e3-5409f9ab0796 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:23.547143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:23.547950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:23.548233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:23.548532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] PATH_INFO: `/v3/projects/3e484fca60594ae8842c9c543d88da20` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:23.549859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e484fca60594ae8842c9c543d88da20)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:23.551834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:23.635294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:23.641354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e484fca60594ae8842c9c543d88da20', 'request_id': 'req-87df1370-9933-4af2-9976-c6e63dba1c63'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:23.646856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e484fca60594ae8842c9c543d88da20', 'request_id': 'req-87df1370-9933-4af2-9976-c6e63dba1c63'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:23.732189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e484fca60594ae8842c9c543d88da20', 'request_id': 'req-87df1370-9933-4af2-9976-c6e63dba1c63'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:23.745346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] The token cache is being invalidate because project 3e484fca60594ae8842c9c543d88da20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:23.747273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:23.747273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-87df1370-9933-4af2-9976-c6e63dba1c63 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:23.748143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1067/2158] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:23 2022] DELETE /identity/v3/projects/3e484fca60594ae8842c9c543d88da20 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:23.753233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1be38a-22ba-47ca-8864-9fd56a9328e4 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:23.753749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1be38a-22ba-47ca-8864-9fd56a9328e4 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:23.754126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1be38a-22ba-47ca-8864-9fd56a9328e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:23.778534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9b1be38a-22ba-47ca-8864-9fd56a9328e4 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:23.780086 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9b1be38a-22ba-47ca-8864-9fd56a9328e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:23.849601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1092/2159] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:23.858009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f5c40b-b5b1-4b71-9ae8-ad34fd16bd7e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:23.861683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f5c40b-b5b1-4b71-9ae8-ad34fd16bd7e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:23.861683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f5c40b-b5b1-4b71-9ae8-ad34fd16bd7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:23.880985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-55f5c40b-b5b1-4b71-9ae8-ad34fd16bd7e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:23.882349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-55f5c40b-b5b1-4b71-9ae8-ad34fd16bd7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:23.891646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1068/2160] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:23.895866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1be38a-22ba-47ca-8864-9fd56a9328e4 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:23.906556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1be38a-22ba-47ca-8864-9fd56a9328e4 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:23.918111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:23.918820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:23.919131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:23.919390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:23.920134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:23.920700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:23.922360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:23.939622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1093/2161] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:20:23.944034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f5c40b-b5b1-4b71-9ae8-ad34fd16bd7e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:23.954876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f5c40b-b5b1-4b71-9ae8-ad34fd16bd7e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:23.967382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:23.968064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:23.970567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:23.970567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:23.970567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:23.972549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:24.093887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1069/2162] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:23 2022] POST /identity/v3/projects => generated 381 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:24.098066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:24.112318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55beb7da-0ea9-4f3d-bb66-9eb074f56ed0 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:24.131085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:24.193983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:24.194887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:24.195170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:24.195447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:24.196145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:24.198253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:24.299198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1094/2163] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:24 2022] POST /identity/v3/users => generated 291 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:24.303922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:24.317377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7584c2bb-7632-46e1-a953-c4eca1104e89 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:24.323764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:24.326813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:24.326813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:24.326813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:24.326813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:24.333354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:24.347293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1070/2164] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:24 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:24.351342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:24.363367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a233af-2368-4fd6-a06d-c0fd6daeffc5 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:24.373295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8877824c-289b-4474-9313-e890d8a260c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:24.379454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8877824c-289b-4474-9313-e890d8a260c7 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:24.379454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8877824c-289b-4474-9313-e890d8a260c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:24.379454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8877824c-289b-4474-9313-e890d8a260c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:24.379454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8877824c-289b-4474-9313-e890d8a260c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:24.379454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8877824c-289b-4474-9313-e890d8a260c7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:24.393838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1095/2165] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:24 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:24.397800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:24.412721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95eb1402-d3a8-42c2-8da8-9a4a99bf64a9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:24.418136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:24.419200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:24.419579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:24.419950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] PATH_INFO: `/v3/projects/3a29420a689e4ace90ead61fe693814e/users/395a6ebaace8421a8a3a5728abe64332/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:24.441767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3a29420a689e4ace90ead61fe693814e, user_id=395a6ebaace8421a8a3a5728abe64332, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:24.445336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:24.521342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1071/2166] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:24 2022] PUT /identity/v3/projects/3a29420a689e4ace90ead61fe693814e/users/395a6ebaace8421a8a3a5728abe64332/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:24.534109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f6310-16cf-46f1-b644-69a474aa174c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:24.534630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f6310-16cf-46f1-b644-69a474aa174c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:24.535030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e5f6310-16cf-46f1-b644-69a474aa174c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:24.589129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0e5f6310-16cf-46f1-b644-69a474aa174c None None] MFA Rules not processed for user `395a6ebaace8421a8a3a5728abe64332`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:24.591252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0e5f6310-16cf-46f1-b644-69a474aa174c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:24.680669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:24.684341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:24.733160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1096/2167] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:24 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:24.738432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f6310-16cf-46f1-b644-69a474aa174c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:24.822069 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f57dc3c-3492-4077-a161-04a6ffbc4555 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:24.834362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:24.835284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:24.835571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:24.835837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] PATH_INFO: `/v3/users/ef6ef54116844f419dc1e50f5c1f37a6` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:24.839252 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ef6ef54116844f419dc1e50f5c1f37a6)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:24.841686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:24.842967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e5f6310-16cf-46f1-b644-69a474aa174c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:24.853690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:24.858598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:24.858936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:24.859225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:24.860232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:24.863173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:24.978204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1097/2168] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:24 2022] POST /identity/v3/projects => generated 383 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:24.979989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:24.994778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:25.067788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ef6ef54116844f419dc1e50f5c1f37a6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:25.103341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772e978b-2c17-4900-ae5f-22ca41b93fa8 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:25.108611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:25.109332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.109857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.109984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.110646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:25.112802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:25.159511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ef6ef54116844f419dc1e50f5c1f37a6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:25.160362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ef6ef54116844f419dc1e50f5c1f37a6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:25.168892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ef6ef54116844f419dc1e50f5c1f37a6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:25.171329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ec612012-3ddd-4a5c-96fc-43d8b626a6c2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ef6ef54116844f419dc1e50f5c1f37a6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:25.187238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=84,keystone:DELETE=12,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:25.193660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1072/2169] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:24 2022] DELETE /identity/v3/users/ef6ef54116844f419dc1e50f5c1f37a6 => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:25.200044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78f320c-949c-4709-b257-448b88383da6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.200591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78f320c-949c-4709-b257-448b88383da6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.201027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78f320c-949c-4709-b257-448b88383da6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.222578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b78f320c-949c-4709-b257-448b88383da6 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:25.224061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b78f320c-949c-4709-b257-448b88383da6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:25.231141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1098/2170] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:24 2022] POST /identity/v3/users => generated 298 bytes in 246 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:25.234582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:25.237286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:25.255401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e94475e-d73f-4a76-99d0-438218ca3690 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:25.327700 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1073/2171] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:20:25.348212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78f320c-949c-4709-b257-448b88383da6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:25.365404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b78f320c-949c-4709-b257-448b88383da6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:25.374028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:25.375078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.375661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.378164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.381073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:25.384457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:25.400490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1099/2172] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:25 2022] GET /identity/v3/roles => generated 1420 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:20:25.411618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:25.485881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbb4990-0a73-467e-9ff3-25e675a6ef01 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:25.501320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:25.502640 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.503048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.503449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] PATH_INFO: `/v3/projects/31939e665c934ca798f44be469f55ac8/users/18ca03095d3a47ec886a4d251a8c11a6/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.544425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] RBAC: Authorizing `identity:create_grant(project_id=31939e665c934ca798f44be469f55ac8, user_id=18ca03095d3a47ec886a4d251a8c11a6, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:25.552801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:25.610337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78f320c-949c-4709-b257-448b88383da6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:25.617655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:25.618271 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.618592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.618925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.642347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:25.643917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:25.646800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1100/2173] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:25 2022] PUT /identity/v3/projects/31939e665c934ca798f44be469f55ac8/users/18ca03095d3a47ec886a4d251a8c11a6/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:25.652286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:25.673767 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1074/2174] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:25.831506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5988e68f-e1a4-428a-bc8d-d5d78201e234 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:25.836562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8e8aee-beef-4577-b7ed-a3c7589cc438 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:25.837257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e8aee-beef-4577-b7ed-a3c7589cc438 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.837639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e8aee-beef-4577-b7ed-a3c7589cc438 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.837869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8e8aee-beef-4577-b7ed-a3c7589cc438 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.839353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e8aee-beef-4577-b7ed-a3c7589cc438 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:25.843325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8e8aee-beef-4577-b7ed-a3c7589cc438 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:25.852869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1101/2175] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:25 2022] GET /identity/v3/roles => generated 1420 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:25.857447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:25.872312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889b3d8-f65e-422d-a380-2a395f4b82c6 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:25.874917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:25.875809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.876088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.876361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] PATH_INFO: `/v3/projects/31939e665c934ca798f44be469f55ac8/users/18ca03095d3a47ec886a4d251a8c11a6/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.878480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=31939e665c934ca798f44be469f55ac8, user_id=18ca03095d3a47ec886a4d251a8c11a6, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:25.881859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:25.897783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1075/2176] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:25 2022] PUT /identity/v3/projects/31939e665c934ca798f44be469f55ac8/users/18ca03095d3a47ec886a4d251a8c11a6/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:25.910605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:25.911925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:25.912432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:25.934618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:25.956001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] MFA Rules not processed for user `18ca03095d3a47ec886a4d251a8c11a6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:25.957315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:26.096479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1102/2177] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:25 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:26.111499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:26.113843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:26.241592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a574f8-0e55-4207-8df3-9689a1e2ac15 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:26.248047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:26.248968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:26.249231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:26.249632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:26.278246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:26.281256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:26.305844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1076/2178] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:26 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 194 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:26.382187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:26.389698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:26.456869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:26.467882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff500408-0c9c-4e04-8573-4009d99de029 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:26.472646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90338ce1-0824-4ad0-a021-b4839d6da493 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:26.473482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90338ce1-0824-4ad0-a021-b4839d6da493 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:26.474019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90338ce1-0824-4ad0-a021-b4839d6da493 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:26.474486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90338ce1-0824-4ad0-a021-b4839d6da493 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:26.476890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-90338ce1-0824-4ad0-a021-b4839d6da493 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:26.497710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:26.553013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90338ce1-0824-4ad0-a021-b4839d6da493 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:26.555776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90338ce1-0824-4ad0-a021-b4839d6da493 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:26.577527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1077/2179] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:26 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:20:26.641422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39c4eb23-b509-4d6b-ad72-68b4c2f2b3d4 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:26.665433 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:26.669356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:26.669668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:26.669954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] PATH_INFO: `/v3/projects/a7383992d769483fae0ef92d4b5ec489` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:26.674270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a7383992d769483fae0ef92d4b5ec489)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:26.681578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:26.924403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a7383992d769483fae0ef92d4b5ec489', 'request_id': 'req-a095e6d7-087f-43c4-bb56-0c0a479e2816'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:27.050378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a7383992d769483fae0ef92d4b5ec489', 'request_id': 'req-a095e6d7-087f-43c4-bb56-0c0a479e2816'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:27.059874 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a7383992d769483fae0ef92d4b5ec489', 'request_id': 'req-a095e6d7-087f-43c4-bb56-0c0a479e2816'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:27.090392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] The token cache is being invalidate because project a7383992d769483fae0ef92d4b5ec489 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:27.090690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:27.091449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a095e6d7-087f-43c4-bb56-0c0a479e2816 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:27.097982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1103/2180] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:26 2022] DELETE /identity/v3/projects/a7383992d769483fae0ef92d4b5ec489 => generated 0 bytes in 718 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:27.099858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915eb48-c09e-48b9-9ec6-962ca051cd2d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.100363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915eb48-c09e-48b9-9ec6-962ca051cd2d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.100466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1915eb48-c09e-48b9-9ec6-962ca051cd2d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:27.151457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1915eb48-c09e-48b9-9ec6-962ca051cd2d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:27.156052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1915eb48-c09e-48b9-9ec6-962ca051cd2d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:27.332352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1078/2181] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:27.334203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1822a-dd4c-4707-a65e-1de4faa3f97a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.334498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1822a-dd4c-4707-a65e-1de4faa3f97a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.334799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1822a-dd4c-4707-a65e-1de4faa3f97a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:27.363902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-68a1822a-dd4c-4707-a65e-1de4faa3f97a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:27.365269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-68a1822a-dd4c-4707-a65e-1de4faa3f97a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:27.373901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1104/2182] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:27.378378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915eb48-c09e-48b9-9ec6-962ca051cd2d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:27.389905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1915eb48-c09e-48b9-9ec6-962ca051cd2d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:27.396301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:27.397089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.397520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.397937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:27.399066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:27.399417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:27.402705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:27.416547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1079/2183] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:20:27.422173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a1822a-dd4c-4707-a65e-1de4faa3f97a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:27.437808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a1822a-dd4c-4707-a65e-1de4faa3f97a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:27.459919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:27.459919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.459919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.459919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:27.460454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:27.460454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:27.493501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:27.584506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1105/2184] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:27 2022] POST /identity/v3/projects => generated 367 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:27.585260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:27.601774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be75986b-5159-4d9e-9dd0-8b787914b086 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:27.725001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:27.726162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.726547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.726993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:27.727821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:27.731114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:27.832778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1080/2185] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:27 2022] POST /identity/v3/users => generated 284 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:27.838543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:27.848657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4a9965-1ec4-42f8-8d05-83b4fd5d6ab3 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:27.854276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:27.857186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.857712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.858246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:27.859319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:27.861197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:27.880675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1106/2186] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:27 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:27.884484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:27.918246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3f5e88-f076-4695-a818-47c62a31d615 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:27.929804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c42f840-9c16-4727-a015-00d9738c5c90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:27.930719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42f840-9c16-4727-a015-00d9738c5c90 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.931216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42f840-9c16-4727-a015-00d9738c5c90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.931765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c42f840-9c16-4727-a015-00d9738c5c90 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:27.932838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42f840-9c16-4727-a015-00d9738c5c90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:27.936819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c42f840-9c16-4727-a015-00d9738c5c90 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:27.960899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1081/2187] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:27 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:27.972790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:27.983848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56db1e76-695e-446e-a451-59fce484f3b2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:27.988396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:27.989353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:27.989633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:27.989906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] PATH_INFO: `/v3/projects/e56b8f071c8c49f4a511578142880b4c/users/a8c5b7d286444676aa1e327dc4a5a9f7/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:28.005221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e56b8f071c8c49f4a511578142880b4c, user_id=a8c5b7d286444676aa1e327dc4a5a9f7, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:28.007955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:28.131503 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1107/2188] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:27 2022] PUT /identity/v3/projects/e56b8f071c8c49f4a511578142880b4c/users/a8c5b7d286444676aa1e327dc4a5a9f7/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:28.143380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd143402-b359-41c6-a890-8e5e01735ed0 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:28.143910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd143402-b359-41c6-a890-8e5e01735ed0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:28.144169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd143402-b359-41c6-a890-8e5e01735ed0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:28.172607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-cd143402-b359-41c6-a890-8e5e01735ed0 None None] MFA Rules not processed for user `a8c5b7d286444676aa1e327dc4a5a9f7`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:28.173734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cd143402-b359-41c6-a890-8e5e01735ed0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:28.183919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:28.260017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1082/2189] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:28 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:29.941250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:29.943170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:30.019011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36a94890-ede4-4df4-a4a3-6eb68570ffe0 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:30.027321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:30.028254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:30.028521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:30.028979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] PATH_INFO: `/v3/users/395a6ebaace8421a8a3a5728abe64332` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:30.030192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] RBAC: Authorizing `identity:delete_user(user_id=395a6ebaace8421a8a3a5728abe64332)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:30.032148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:30.194537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '395a6ebaace8421a8a3a5728abe64332'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:30.267517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '395a6ebaace8421a8a3a5728abe64332'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:30.275121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '395a6ebaace8421a8a3a5728abe64332'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:30.275891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '395a6ebaace8421a8a3a5728abe64332'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:30.276533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '395a6ebaace8421a8a3a5728abe64332'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:30.296228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1108/2190] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:29 2022] DELETE /identity/v3/users/395a6ebaace8421a8a3a5728abe64332 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:30.301844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:30.301844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:30.301844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:30.319959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:30.321262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:30.388924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1083/2191] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:30.395881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:30.397465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:30.444442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:INSERT=3,keystone:SELECT=87,keystone:DELETE=10 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:30.473193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5042c5e0-896c-4f3c-8822-50be292fc468 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:30.478373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:30.479915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:30.480253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:30.480541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:30.491967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:30.493695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:30.505010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1109/2192] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:31.132478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:31.133916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:31.151925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7196640f-6688-4cf7-a860-625bac787c54 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:31.156990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:31.158309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:31.158707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:31.159022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] PATH_INFO: `/v3/projects/3a29420a689e4ace90ead61fe693814e` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:31.163033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3a29420a689e4ace90ead61fe693814e)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:31.163033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:31.292869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3a29420a689e4ace90ead61fe693814e', 'request_id': 'req-a12a9eb6-db53-4f28-a97e-673ff2e4e018'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.299655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3a29420a689e4ace90ead61fe693814e', 'request_id': 'req-a12a9eb6-db53-4f28-a97e-673ff2e4e018'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.382549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3a29420a689e4ace90ead61fe693814e', 'request_id': 'req-a12a9eb6-db53-4f28-a97e-673ff2e4e018'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.397707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] The token cache is being invalidate because project 3a29420a689e4ace90ead61fe693814e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:31.399656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.399656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.401372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1084/2193] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:31 2022] DELETE /identity/v3/projects/3a29420a689e4ace90ead61fe693814e => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:31.406087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:31.419609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba363994-306f-46a4-abde-5136c85ad543 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:31.489354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:31.491519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:31.492015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:31.492499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] PATH_INFO: `/v3/users/18ca03095d3a47ec886a4d251a8c11a6` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:31.494003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] RBAC: Authorizing `identity:delete_user(user_id=18ca03095d3a47ec886a4d251a8c11a6)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:31.497007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:31.796743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '18ca03095d3a47ec886a4d251a8c11a6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.850598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '18ca03095d3a47ec886a4d251a8c11a6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.858382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '18ca03095d3a47ec886a4d251a8c11a6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.859132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '18ca03095d3a47ec886a4d251a8c11a6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.859786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4ee3603d-8679-44cf-a821-2ab262347d06 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '18ca03095d3a47ec886a4d251a8c11a6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:31.882465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1110/2194] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:31 2022] DELETE /identity/v3/users/18ca03095d3a47ec886a4d251a8c11a6 => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:32.659601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:32.663663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:32.677569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12a9eb6-db53-4f28-a97e-673ff2e4e018 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:32.720130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:32.752932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:32.753798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:32.754079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:32.754353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] PATH_INFO: `/v3/projects/31939e665c934ca798f44be469f55ac8` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:32.755578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] RBAC: Authorizing `identity:delete_project(project_id=31939e665c934ca798f44be469f55ac8)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:32.757677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:32.909262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '31939e665c934ca798f44be469f55ac8', 'request_id': 'req-03a739df-396f-48ed-ae60-46f65937bc3e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:32.925905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '31939e665c934ca798f44be469f55ac8', 'request_id': 'req-03a739df-396f-48ed-ae60-46f65937bc3e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:32.987905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '31939e665c934ca798f44be469f55ac8', 'request_id': 'req-03a739df-396f-48ed-ae60-46f65937bc3e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:33.001595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] The token cache is being invalidate because project 31939e665c934ca798f44be469f55ac8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:33.001933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:33.002527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-03a739df-396f-48ed-ae60-46f65937bc3e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:33.004475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1085/2195] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:32 2022] DELETE /identity/v3/projects/31939e665c934ca798f44be469f55ac8 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:33.009636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9c69b7-0809-4527-8bbb-8dd89b502168 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.010810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9c69b7-0809-4527-8bbb-8dd89b502168 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.010911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9c69b7-0809-4527-8bbb-8dd89b502168 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.047559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-7f9c69b7-0809-4527-8bbb-8dd89b502168 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:33.058445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7f9c69b7-0809-4527-8bbb-8dd89b502168 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:33.164374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:33.180606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1111/2196] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:33.189273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03cea3-e923-48ca-aff3-d80e444df579 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.189827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03cea3-e923-48ca-aff3-d80e444df579 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.190217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e03cea3-e923-48ca-aff3-d80e444df579 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.213450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-3e03cea3-e923-48ca-aff3-d80e444df579 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:33.214850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3e03cea3-e923-48ca-aff3-d80e444df579 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:33.228294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1086/2197] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:33.231250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9c69b7-0809-4527-8bbb-8dd89b502168 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:33.240808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9c69b7-0809-4527-8bbb-8dd89b502168 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:33.246150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:33.246917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.247283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.247628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.248435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:33.248814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:33.250551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:33.276710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1112/2198] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:20:33.277639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03cea3-e923-48ca-aff3-d80e444df579 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:33.288212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e03cea3-e923-48ca-aff3-d80e444df579 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:33.293274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:33.294153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.294443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.294756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.295960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:33.297798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:33.401741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1087/2199] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:33 2022] POST /identity/v3/projects => generated 371 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:33.405875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:33.416826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d05c8e-88b6-45da-89b3-59f603450154 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:33.477368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:33.478094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.478411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.479263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.480118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:33.482080 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:33.623696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1113/2200] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:33 2022] POST /identity/v3/users => generated 286 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:33.627764 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:33.639419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a51870d-3f30-41d4-8acb-db8555a53e2e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:33.644892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:33.645627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.645933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.646212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.647091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:33.648882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:33.661316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1088/2201] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:33 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:33.665464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:33.678809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f3c0c7d-081a-481e-b1d2-204aff4e081e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:33.685454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bbfbef-4f34-4d48-9c87-39fa2ed0f91c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:33.686174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bbfbef-4f34-4d48-9c87-39fa2ed0f91c admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.687034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bbfbef-4f34-4d48-9c87-39fa2ed0f91c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.687318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bbfbef-4f34-4d48-9c87-39fa2ed0f91c admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.688109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bbfbef-4f34-4d48-9c87-39fa2ed0f91c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:33.689957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bbfbef-4f34-4d48-9c87-39fa2ed0f91c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:33.704452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1114/2202] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:33 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:33.705665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:33.721073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38c956f1-f31e-4d3b-9283-433b80a8852e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:33.729384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:33.734575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.734575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.734575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] PATH_INFO: `/v3/projects/276b03a2eade4a3b8eab700880894c99/users/c43b346117de472e86c4f0fe4f7b1fa9/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.750076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] RBAC: Authorizing `identity:create_grant(project_id=276b03a2eade4a3b8eab700880894c99, user_id=c43b346117de472e86c4f0fe4f7b1fa9, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:33.753410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:33.822951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1089/2203] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:33 2022] PUT /identity/v3/projects/276b03a2eade4a3b8eab700880894c99/users/c43b346117de472e86c4f0fe4f7b1fa9/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:33.834643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44889fe-016b-445b-baf9-74b000b8806d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:33.834958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44889fe-016b-445b-baf9-74b000b8806d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:33.835287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44889fe-016b-445b-baf9-74b000b8806d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:33.871699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d44889fe-016b-445b-baf9-74b000b8806d None None] MFA Rules not processed for user `c43b346117de472e86c4f0fe4f7b1fa9`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:33.873317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d44889fe-016b-445b-baf9-74b000b8806d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:33.948252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1115/2204] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:33 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:33.959671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:33.961555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:34.032751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d345ed35-7e8e-46e9-b464-3ac10ea44a4a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:34.036691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67a91574-0c9b-43da-8f4e-1472f9ed0f7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:34.037351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a91574-0c9b-43da-8f4e-1472f9ed0f7d service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:34.037623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a91574-0c9b-43da-8f4e-1472f9ed0f7d service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:34.037905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67a91574-0c9b-43da-8f4e-1472f9ed0f7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:34.047907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a91574-0c9b-43da-8f4e-1472f9ed0f7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:34.049560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67a91574-0c9b-43da-8f4e-1472f9ed0f7d service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:34.058955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1090/2205] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:33 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:34.079114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44889fe-016b-445b-baf9-74b000b8806d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:34.154161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:34.162614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d44889fe-016b-445b-baf9-74b000b8806d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:34.168382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:34.169497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:34.169927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:34.170182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] PATH_INFO: `/v3/users/c43b346117de472e86c4f0fe4f7b1fa9` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:34.171729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c43b346117de472e86c4f0fe4f7b1fa9)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:34.173653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:34.392720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c43b346117de472e86c4f0fe4f7b1fa9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:34.462636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c43b346117de472e86c4f0fe4f7b1fa9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:34.472263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c43b346117de472e86c4f0fe4f7b1fa9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:34.473055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c43b346117de472e86c4f0fe4f7b1fa9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:34.473754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c43b346117de472e86c4f0fe4f7b1fa9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:34.495843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1116/2206] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:34 2022] DELETE /identity/v3/users/c43b346117de472e86c4f0fe4f7b1fa9 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:20:34.500096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:34.500387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:34.500769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:34.508953 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:34.521842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:34.523533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:34.570855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:34.575903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:34.613649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1091/2207] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:20:34.629301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:34.630666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:34.653484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b5f1e4-17bc-45d0-93a2-29ba095da4b8 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:34.657149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533391ff-7dff-432e-994f-0d91a96f1943 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:34.657905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533391ff-7dff-432e-994f-0d91a96f1943 service glance] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:34.658196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533391ff-7dff-432e-994f-0d91a96f1943 service glance] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:34.658478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533391ff-7dff-432e-994f-0d91a96f1943 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:34.660318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-533391ff-7dff-432e-994f-0d91a96f1943 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:34.700359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc303f5-f17f-4ed3-b5d3-a52f78fef8c5 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:34.705254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:34.706132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:34.706406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:34.706674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:34.723528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:34.724704 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:34.735738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533391ff-7dff-432e-994f-0d91a96f1943 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:34.737200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-533391ff-7dff-432e-994f-0d91a96f1943 service glance] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:34.737708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1092/2208] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:20:34.756414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1117/2209] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:34 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:20:35.179678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:35.181307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:35.196566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95315cb0-ab93-410e-af85-7f2692ee2c14 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:35.201827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:35.202822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:35.203089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:35.203513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] PATH_INFO: `/v3/projects/276b03a2eade4a3b8eab700880894c99` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:35.205797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] RBAC: Authorizing `identity:delete_project(project_id=276b03a2eade4a3b8eab700880894c99)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:35.207762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:35.380018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '276b03a2eade4a3b8eab700880894c99', 'request_id': 'req-8cab6fb3-15cf-4276-8886-9a3e8894c66e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:35.387631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '276b03a2eade4a3b8eab700880894c99', 'request_id': 'req-8cab6fb3-15cf-4276-8886-9a3e8894c66e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:35.469543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '276b03a2eade4a3b8eab700880894c99', 'request_id': 'req-8cab6fb3-15cf-4276-8886-9a3e8894c66e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:35.490585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] The token cache is being invalidate because project 276b03a2eade4a3b8eab700880894c99 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:35.490832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:35.492477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8cab6fb3-15cf-4276-8886-9a3e8894c66e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:35.495825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1093/2210] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:35 2022] DELETE /identity/v3/projects/276b03a2eade4a3b8eab700880894c99 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:35.500648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22498369-c711-4361-87db-57496e79d52d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:35.503033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22498369-c711-4361-87db-57496e79d52d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:35.503033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22498369-c711-4361-87db-57496e79d52d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:35.520508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-22498369-c711-4361-87db-57496e79d52d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:35.521714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-22498369-c711-4361-87db-57496e79d52d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:35.563394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:35.616182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1118/2211] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:35.625067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229447-ce81-482a-bd2c-26e99fdd6915 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:35.625558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229447-ce81-482a-bd2c-26e99fdd6915 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:35.625925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff229447-ce81-482a-bd2c-26e99fdd6915 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:35.660789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ff229447-ce81-482a-bd2c-26e99fdd6915 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:35.661083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ff229447-ce81-482a-bd2c-26e99fdd6915 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:35.674153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1094/2212] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:35.680518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22498369-c711-4361-87db-57496e79d52d None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:35.704243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22498369-c711-4361-87db-57496e79d52d None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:35.729783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:35.735320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:35.735320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:35.735320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:35.735320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:35.735320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:35.740314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:35.766803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1119/2213] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:20:35.770961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229447-ce81-482a-bd2c-26e99fdd6915 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:35.784594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff229447-ce81-482a-bd2c-26e99fdd6915 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:35.798371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:35.799474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:35.800548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:35.801009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:35.802957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:35.806620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:35.939729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1095/2214] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:35 2022] POST /identity/v3/projects => generated 385 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:35.940848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:35.951634 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61599d53-cb0e-4b32-acee-62d276ce80a2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:36.006759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:36.007446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:36.007747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:36.008005 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:36.008685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:36.011324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:36.097491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1120/2215] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:35 2022] POST /identity/v3/users => generated 293 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:36.101457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:36.108983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:36.112657 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0a675b-8960-42c7-a7ef-4fb6eb41be89 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:36.117505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:36.118184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:36.118474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:36.120535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:36.121307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:36.123555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:36.134566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1096/2216] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:36 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:20:36.143276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:36.149417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15e36592-d8c4-41a9-9544-e8e7db3ce0a5 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:36.154289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efdb181-2f49-4341-b45e-f1d59b534f94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:36.154973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efdb181-2f49-4341-b45e-f1d59b534f94 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:36.155261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efdb181-2f49-4341-b45e-f1d59b534f94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:36.155543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efdb181-2f49-4341-b45e-f1d59b534f94 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:36.156270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efdb181-2f49-4341-b45e-f1d59b534f94 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:36.157956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8efdb181-2f49-4341-b45e-f1d59b534f94 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:36.168121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1121/2217] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:36.179496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:36.190099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db806cf-4077-4ef1-8e78-9da8798f8892 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:36.195753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:36.196629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:36.196912 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:36.197796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] PATH_INFO: `/v3/projects/69620351904f497fbe13d5f6c2d51994/users/f0a5f78e3eb24219a9e5e4acecb17707/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:36.212253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=69620351904f497fbe13d5f6c2d51994, user_id=f0a5f78e3eb24219a9e5e4acecb17707, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:36.215144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:36.296895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1097/2218] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:36 2022] PUT /identity/v3/projects/69620351904f497fbe13d5f6c2d51994/users/f0a5f78e3eb24219a9e5e4acecb17707/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:36.316423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718ae1c-87ce-4060-8a53-6f5af2f257c1 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:36.316423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718ae1c-87ce-4060-8a53-6f5af2f257c1 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:36.316423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b718ae1c-87ce-4060-8a53-6f5af2f257c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:36.338440 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b718ae1c-87ce-4060-8a53-6f5af2f257c1 None None] MFA Rules not processed for user `f0a5f78e3eb24219a9e5e4acecb17707`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:36.339677 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b718ae1c-87ce-4060-8a53-6f5af2f257c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:36.404500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1122/2219] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:36 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:36.417926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:36.419796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:36.490388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d57c06b4-4aaa-46c4-9025-fdd64952dba5 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:36.496977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5f1cbd-6f16-4f5a-b63c-4310888affa8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:36.497648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f1cbd-6f16-4f5a-b63c-4310888affa8 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:36.497932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f1cbd-6f16-4f5a-b63c-4310888affa8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:36.498211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5f1cbd-6f16-4f5a-b63c-4310888affa8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:36.507609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5f1cbd-6f16-4f5a-b63c-4310888affa8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:36.510411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5f1cbd-6f16-4f5a-b63c-4310888affa8 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:36.524851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1098/2220] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:36 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:39.750541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718ae1c-87ce-4060-8a53-6f5af2f257c1 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:39.781919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:39.825114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b718ae1c-87ce-4060-8a53-6f5af2f257c1 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:39.830219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:39.832470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:39.832470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:39.832470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] PATH_INFO: `/v3/users/f0a5f78e3eb24219a9e5e4acecb17707` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:39.832842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0a5f78e3eb24219a9e5e4acecb17707)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:39.834754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:40.024684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0a5f78e3eb24219a9e5e4acecb17707'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:40.073664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0a5f78e3eb24219a9e5e4acecb17707'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:40.081701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0a5f78e3eb24219a9e5e4acecb17707'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:40.082383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0a5f78e3eb24219a9e5e4acecb17707'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:40.083196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0a5f78e3eb24219a9e5e4acecb17707'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:40.104428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1123/2221] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:39 2022] DELETE /identity/v3/users/f0a5f78e3eb24219a9e5e4acecb17707 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:40.109361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:40.109906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:40.109906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:40.129861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:40.136994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:40.210065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1099/2222] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:20:40.220369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:40.221800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:40.301643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62482ca3-cbbe-446c-952f-9067b98630f2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:40.305362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1b93f1e-2878-4b86-8668-45ca087437f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:40.306020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b93f1e-2878-4b86-8668-45ca087437f1 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:40.306301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b93f1e-2878-4b86-8668-45ca087437f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:40.306733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1b93f1e-2878-4b86-8668-45ca087437f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:40.317095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b93f1e-2878-4b86-8668-45ca087437f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:40.318544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1b93f1e-2878-4b86-8668-45ca087437f1 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:40.330387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1124/2223] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:40.873516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:40.875644 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:40.923762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba1d43bf-d8af-4d8a-a1d7-a3eeb2d6e02d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:40.930879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:40.932596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:40.932997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:40.933611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] PATH_INFO: `/v3/projects/69620351904f497fbe13d5f6c2d51994` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:40.935159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69620351904f497fbe13d5f6c2d51994)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:40.938027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:41.065103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69620351904f497fbe13d5f6c2d51994', 'request_id': 'req-0f282e44-f6db-4d63-902b-573342c285f7'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:41.075254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69620351904f497fbe13d5f6c2d51994', 'request_id': 'req-0f282e44-f6db-4d63-902b-573342c285f7'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:41.130606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69620351904f497fbe13d5f6c2d51994', 'request_id': 'req-0f282e44-f6db-4d63-902b-573342c285f7'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:41.132295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:41.152258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] The token cache is being invalidate because project 69620351904f497fbe13d5f6c2d51994 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:41.152648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:41.153982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-0f282e44-f6db-4d63-902b-573342c285f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:41.156746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1100/2224] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:40 2022] DELETE /identity/v3/projects/69620351904f497fbe13d5f6c2d51994 => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:41.162557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d0da70-5799-44f6-a9d8-a469106f7e23 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.163100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d0da70-5799-44f6-a9d8-a469106f7e23 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.163258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d0da70-5799-44f6-a9d8-a469106f7e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.184709 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-90d0da70-5799-44f6-a9d8-a469106f7e23 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:41.186507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-90d0da70-5799-44f6-a9d8-a469106f7e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:41.248937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:41.276170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1125/2225] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:41.285988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53853b6c-067e-4ee6-8a2e-ddb76976b9de None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.286491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53853b6c-067e-4ee6-8a2e-ddb76976b9de None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.286799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53853b6c-067e-4ee6-8a2e-ddb76976b9de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.333243 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-53853b6c-067e-4ee6-8a2e-ddb76976b9de None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:41.334703 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-53853b6c-067e-4ee6-8a2e-ddb76976b9de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:41.367157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1101/2226] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:41.368701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d0da70-5799-44f6-a9d8-a469106f7e23 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:41.383847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d0da70-5799-44f6-a9d8-a469106f7e23 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:41.413977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:41.415154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.415697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.416520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.417836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:41.418323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:41.420186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:41.442832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1126/2227] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:20:41.447826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53853b6c-067e-4ee6-8a2e-ddb76976b9de None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:41.457674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53853b6c-067e-4ee6-8a2e-ddb76976b9de None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:41.464997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:41.465673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.465961 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.466301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.467186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:41.469123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:41.552922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1102/2228] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:41 2022] POST /identity/v3/projects => generated 383 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:41.558537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:41.571894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3414ec8c-ebcd-43bf-9a76-56ae172f086e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:41.640902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:41.641492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.641787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.642273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.642916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:41.644892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:41.746808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1127/2229] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:41 2022] POST /identity/v3/users => generated 292 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:41.750417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:41.762347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67afe08-7115-4108-9c8c-721f6c12d646 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:41.768155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:41.768837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.769124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.769402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.770142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:41.771856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:41.786516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1103/2230] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:41 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:41.799220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:41.810603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c963331-82e2-4866-8293-2f08972c0477 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:41.818478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07f458ca-c29a-473a-9fc4-8d7f8bb47aeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:41.819696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f458ca-c29a-473a-9fc4-8d7f8bb47aeb admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.820300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f458ca-c29a-473a-9fc4-8d7f8bb47aeb admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.821332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07f458ca-c29a-473a-9fc4-8d7f8bb47aeb admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.823975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f458ca-c29a-473a-9fc4-8d7f8bb47aeb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:41.827477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07f458ca-c29a-473a-9fc4-8d7f8bb47aeb admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:41.846465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1128/2231] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:41 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:41.851150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:41.863474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a6109d-ac1e-4010-8a61-a30c9e9856b9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:41.889842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:41.891146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:41.891372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:41.891739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] PATH_INFO: `/v3/projects/a37bbdf2bd2f4392ab2ddb60d6b6ad7c/users/c72a81dcca3145fe9f9623e49baba829/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:41.957292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a37bbdf2bd2f4392ab2ddb60d6b6ad7c, user_id=c72a81dcca3145fe9f9623e49baba829, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:41.960293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:42.139621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1104/2232] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:41 2022] PUT /identity/v3/projects/a37bbdf2bd2f4392ab2ddb60d6b6ad7c/users/c72a81dcca3145fe9f9623e49baba829/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:42.148439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee07d106-16af-4dec-94f3-441d09bf2c7e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:42.149293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee07d106-16af-4dec-94f3-441d09bf2c7e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:42.149948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee07d106-16af-4dec-94f3-441d09bf2c7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:42.218234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ee07d106-16af-4dec-94f3-441d09bf2c7e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:42.218234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ee07d106-16af-4dec-94f3-441d09bf2c7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:42.292582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1129/2233] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:42.303601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:42.305601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:42.374589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e24e7c-e7a2-4f76-b4fe-2d9a54fce013 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:42.378782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:42.380790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:42.380790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:42.380790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:42.391621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:42.393223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:42.409395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1105/2234] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:20:52.282971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=73,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:52.303430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de85f1-16d8-4112-b282-1e1d25e02fd2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:52.303793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de85f1-16d8-4112-b282-1e1d25e02fd2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:52.304152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de85f1-16d8-4112-b282-1e1d25e02fd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:52.335034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d6de85f1-16d8-4112-b282-1e1d25e02fd2 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:52.336261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d6de85f1-16d8-4112-b282-1e1d25e02fd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:52.398416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1130/2235] 10.222.0.27 () {58 vars in 967 bytes} [Thu Jul 14 12:20:52 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:52.400143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=56,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:52.413969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:52.418924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:52.525058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7538d226-e6f3-4576-957b-d30c112636e1 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:52.531444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:52.532195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:52.532487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:52.532770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:52.541813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:52.543406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:52.552680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1106/2236] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:20:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:20:52.935771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770626c4-c118-4d12-9efa-068bfa4423ac None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:52.940791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770626c4-c118-4d12-9efa-068bfa4423ac None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:52.940791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-770626c4-c118-4d12-9efa-068bfa4423ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:52.966345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-770626c4-c118-4d12-9efa-068bfa4423ac None None] MFA Rules not processed for user `c72a81dcca3145fe9f9623e49baba829`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:52.967675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-770626c4-c118-4d12-9efa-068bfa4423ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:53.049681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1131/2237] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:52 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:53.054292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:53.065713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c567c31-f7df-4641-a64b-e1643c51ed8c service swift] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:53.070417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:53.071115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:53.071406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:53.071673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:53.072593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:53.074560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:53.184507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1107/2238] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:53 2022] POST /identity/v3/projects => generated 385 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:53.193684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770626c4-c118-4d12-9efa-068bfa4423ac None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:53.203894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-770626c4-c118-4d12-9efa-068bfa4423ac None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:53.322174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:53.323151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:53.323553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:53.323944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:53.324942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:53.327150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:53.492079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1132/2239] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:53 2022] POST /identity/v3/users => generated 299 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:53.497950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:53.512490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a44fe54-8c6b-448c-b5c2-8925a167256c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:53.518137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:53.519062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:53.519586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:53.520067 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:53.521058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:53.523432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:53.540088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1108/2240] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:53 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:53.545538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:53.558260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f34c57-5dbe-47e2-b42f-a03ff8e9dd4e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:53.566809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:53.568322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:53.571217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:53.575506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] PATH_INFO: `/v3/projects/08c5082fed204967b2dd11bfbbfdce26/users/1e473e1674374ff583113b2c38fc7752/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:53.595785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08c5082fed204967b2dd11bfbbfdce26, user_id=1e473e1674374ff583113b2c38fc7752, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:53.598995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:53.738623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:53.745317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1133/2241] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:53 2022] PUT /identity/v3/projects/08c5082fed204967b2dd11bfbbfdce26/users/1e473e1674374ff583113b2c38fc7752/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:53.749418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:53.760941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19c8754-9b27-4a38-b26e-1a0190989cf2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:53.837782 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:53.838578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:53.838753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:53.839237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:53.839998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:53.841795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:53.853781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1109/2242] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:53 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:53.861591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:53.880108 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c893f8-c04f-4dd5-8885-44c7121e2aa8 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:53.887162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:53.888376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:53.888758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:53.889134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] PATH_INFO: `/v3/projects/08c5082fed204967b2dd11bfbbfdce26/users/1e473e1674374ff583113b2c38fc7752/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:53.892391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=08c5082fed204967b2dd11bfbbfdce26, user_id=1e473e1674374ff583113b2c38fc7752, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:53.895597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:53.910388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1134/2243] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:53 2022] PUT /identity/v3/projects/08c5082fed204967b2dd11bfbbfdce26/users/1e473e1674374ff583113b2c38fc7752/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:56.694221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:56.697291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:56.712847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febbb993-f37b-4d11-a48e-cdc4770415eb admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:56.788740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:56.799928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:56.800918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:56.801267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:56.801558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] PATH_INFO: `/v3/users/3ccee18a56e44ccebfa14d264deb219b` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:56.803104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3ccee18a56e44ccebfa14d264deb219b)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:56.805279 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:57.010720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3ccee18a56e44ccebfa14d264deb219b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:57.121343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3ccee18a56e44ccebfa14d264deb219b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:57.121343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3ccee18a56e44ccebfa14d264deb219b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:57.131516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3ccee18a56e44ccebfa14d264deb219b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:57.137653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b02390d0-a006-4ffd-a7db-aad9753ef515 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3ccee18a56e44ccebfa14d264deb219b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:57.160016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1110/2244] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:20:56 2022] DELETE /identity/v3/users/3ccee18a56e44ccebfa14d264deb219b => generated 0 bytes in 466 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:57.647712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:57.650204 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:57.673605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a69857-377e-40de-84f6-e901cd179d31 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:57.800766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:57.801657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:57.801939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:57.802281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] PATH_INFO: `/v3/projects/8bf2b73c991f489d915d7124b5d24564` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:57.803823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=8bf2b73c991f489d915d7124b5d24564)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:57.807361 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:57.956828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8bf2b73c991f489d915d7124b5d24564', 'request_id': 'req-278c4528-41de-42ad-87ea-f2ce003206ee'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:58.027785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8bf2b73c991f489d915d7124b5d24564', 'request_id': 'req-278c4528-41de-42ad-87ea-f2ce003206ee'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:58.038515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8bf2b73c991f489d915d7124b5d24564', 'request_id': 'req-278c4528-41de-42ad-87ea-f2ce003206ee'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:58.053465 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] The token cache is being invalidate because project 8bf2b73c991f489d915d7124b5d24564 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:20:58.053759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:58.054598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-278c4528-41de-42ad-87ea-f2ce003206ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:20:58.056616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1135/2245] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:20:57 2022] DELETE /identity/v3/projects/8bf2b73c991f489d915d7124b5d24564 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:58.061552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef8821-05f2-49ad-ba49-5a90ed8481bb None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.062051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef8821-05f2-49ad-ba49-5a90ed8481bb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.062272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02ef8821-05f2-49ad-ba49-5a90ed8481bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.081435 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-02ef8821-05f2-49ad-ba49-5a90ed8481bb None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:58.082688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-02ef8821-05f2-49ad-ba49-5a90ed8481bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:58.158346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1111/2246] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:58.165783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76237cd3-f6ba-4e7a-ab43-f16c777a3bff None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.166351 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76237cd3-f6ba-4e7a-ab43-f16c777a3bff None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.167020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76237cd3-f6ba-4e7a-ab43-f16c777a3bff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.199926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-76237cd3-f6ba-4e7a-ab43-f16c777a3bff None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:58.202211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-76237cd3-f6ba-4e7a-ab43-f16c777a3bff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:58.219963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1136/2247] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:58.226269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ef8821-05f2-49ad-ba49-5a90ed8481bb None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:58.234436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02ef8821-05f2-49ad-ba49-5a90ed8481bb None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:58.242834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:58.243924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.244387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.244963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.246603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:20:58.247796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:58.249957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:58.278128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1112/2248] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:20:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:20:58.278359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76237cd3-f6ba-4e7a-ab43-f16c777a3bff None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:58.291522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76237cd3-f6ba-4e7a-ab43-f16c777a3bff None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:58.304660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:58.305452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.305806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.306370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.307188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:58.309399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:58.409318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1137/2249] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:20:58 2022] POST /identity/v3/projects => generated 373 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:20:58.413498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:58.424112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-629c0e18-2841-434b-bf31-970b19dd97d4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:58.495672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:58.496252 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.496635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.497054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.497863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:58.500019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:58.507236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:58.575407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1113/2250] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:20:58 2022] POST /identity/v3/users => generated 287 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:20:58.576736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:58.587371 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620caea3-20f9-4f08-8cb1-fa07acc103fa admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:58.593064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:58.599320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.599320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.599320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.599320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:58.599320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:58.611655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1138/2251] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:58 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:58.614353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:58.629407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee44cdc-fc1d-443b-9d5c-d63e360fc5b2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:58.635547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e8c905-514e-43fa-961b-e2462a06f06d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:58.636222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8c905-514e-43fa-961b-e2462a06f06d admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.636509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8c905-514e-43fa-961b-e2462a06f06d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.636794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e8c905-514e-43fa-961b-e2462a06f06d admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.637591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e8c905-514e-43fa-961b-e2462a06f06d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:58.639513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e8c905-514e-43fa-961b-e2462a06f06d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:58.655212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1114/2252] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:20:58 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:20:58.656396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:58.667238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf5c201-0643-40d1-a861-1c67b788d110 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:58.672303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:58.673416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.673899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.674308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] PATH_INFO: `/v3/projects/206723edeb144547882daa902124ed7c/users/ca8ddc2748da4b05bc6d7ee16ee3db24/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.692688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=206723edeb144547882daa902124ed7c, user_id=ca8ddc2748da4b05bc6d7ee16ee3db24, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:58.696306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:58.778906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1139/2253] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:20:58 2022] PUT /identity/v3/projects/206723edeb144547882daa902124ed7c/users/ca8ddc2748da4b05bc6d7ee16ee3db24/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:20:58.791178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0c244-061b-4137-ae8b-ecfd31963af9 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.791622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0c244-061b-4137-ae8b-ecfd31963af9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.792347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c0c244-061b-4137-ae8b-ecfd31963af9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:58.821581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-55c0c244-061b-4137-ae8b-ecfd31963af9 None None] MFA Rules not processed for user `ca8ddc2748da4b05bc6d7ee16ee3db24`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:20:58.823875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-55c0c244-061b-4137-ae8b-ecfd31963af9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:58.910212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1115/2254] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:20:58 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:20:58.923686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:20:58.925549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:20:58.948856 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:20:58.994860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28a73bb-5296-4d30-9356-1779224353aa admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:20:58.998586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:20:58.999340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:20:58.999623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:20:58.999899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:20:59.008816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:20:59.010947 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:20:59.020061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1140/2255] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:20:58 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:00.713756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416363e8-5aea-45e8-9109-a50d3699569b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:00.714343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416363e8-5aea-45e8-9109-a50d3699569b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:00.714741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416363e8-5aea-45e8-9109-a50d3699569b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:00.754924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-416363e8-5aea-45e8-9109-a50d3699569b None None] MFA Rules not processed for user `1e473e1674374ff583113b2c38fc7752`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:00.756211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-416363e8-5aea-45e8-9109-a50d3699569b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:00.831135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1116/2256] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:00 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:00.843701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:00.845382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:00.898790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba2e2bf-0159-4a9b-b53a-cf0e51bf93b8 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:00.899168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba2e2bf-0159-4a9b-b53a-cf0e51bf93b8 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:00.899564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ba2e2bf-0159-4a9b-b53a-cf0e51bf93b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:00.910356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c354d5f4-0bc2-4e0d-b8a1-b76975dd0cd8 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:00.914508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:00.915287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:00.916128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:00.916405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:00.918359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:00.920536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4ba2e2bf-0159-4a9b-b53a-cf0e51bf93b8 None None] MFA Rules not processed for user `ca8ddc2748da4b05bc6d7ee16ee3db24`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:00.921965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4ba2e2bf-0159-4a9b-b53a-cf0e51bf93b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:00.933003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1117/2257] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:00 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:21:00.982568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:00.984249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:00.994466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1141/2258] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:00 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:01.229453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ae2614-4371-4092-ab12-f0cdd6380d45 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:01.230397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ae2614-4371-4092-ab12-f0cdd6380d45 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:01.230397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ae2614-4371-4092-ab12-f0cdd6380d45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:01.250414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-19ae2614-4371-4092-ab12-f0cdd6380d45 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:01.251685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-19ae2614-4371-4092-ab12-f0cdd6380d45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:01.301415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:01.316638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1118/2259] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:21:01 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:01.331327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:01.333828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:01.411169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce62411b-76aa-4ebc-85a2-34435f139435 service glance] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:01.416956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b519ba7a-c86e-4cad-ae9b-45398e7dfb01 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:01.417649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519ba7a-c86e-4cad-ae9b-45398e7dfb01 service swift] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:01.417929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519ba7a-c86e-4cad-ae9b-45398e7dfb01 service swift] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:01.418202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b519ba7a-c86e-4cad-ae9b-45398e7dfb01 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:01.426949 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b519ba7a-c86e-4cad-ae9b-45398e7dfb01 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:01.428632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b519ba7a-c86e-4cad-ae9b-45398e7dfb01 service swift] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:01.434096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:01.438316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1142/2260] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:21:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:21:08.783904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:08.785911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:08.787335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:08.820854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] MFA Rules not processed for user `0d302e57946c42d799e9bf120873eba6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:08.823010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:08.933020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1119/2261] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:08 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:14.945750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc28825-b855-4f2e-ac95-eb6e266d96c7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:14.946336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc28825-b855-4f2e-ac95-eb6e266d96c7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:14.946336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc28825-b855-4f2e-ac95-eb6e266d96c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:14.964144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9bc28825-b855-4f2e-ac95-eb6e266d96c7 None None] MFA Rules not processed for user `c72a81dcca3145fe9f9623e49baba829`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:14.965209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9bc28825-b855-4f2e-ac95-eb6e266d96c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:14.972150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1143/2262] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:17.360321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:17.362037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:17.422541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23519e3e-bd86-40f6-87d9-d6bb902852db None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:17.426412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:17.427099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:17.427386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:17.427663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:17.429465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:17.496788 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:17.498241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:17.508426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1120/2263] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:17 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:19.258468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:19.259652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:19.259652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:19.285077 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:19.287408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:19.297651 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1144/2264] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:21:19 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:19.320677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:19.339064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a446b151-e96b-43c8-9b0f-b0ca8a534d77 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:19.345594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af8a797-c573-44ad-8c49-12e8b52a9858 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:19.346304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af8a797-c573-44ad-8c49-12e8b52a9858 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:19.346588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af8a797-c573-44ad-8c49-12e8b52a9858 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:19.346895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af8a797-c573-44ad-8c49-12e8b52a9858 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:19.360021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af8a797-c573-44ad-8c49-12e8b52a9858 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:19.361649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af8a797-c573-44ad-8c49-12e8b52a9858 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:19.377733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1121/2265] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:21:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:21:19.865251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:19.869046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:20.001835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5286ff53-ec75-4232-8f2f-bb46c5b3d184 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:20.008123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:20.010450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:20.010966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:20.012508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] PATH_INFO: `/v3/users/a8c5b7d286444676aa1e327dc4a5a9f7` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:20.014581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8c5b7d286444676aa1e327dc4a5a9f7)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:20.017894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:20.235680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8c5b7d286444676aa1e327dc4a5a9f7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:20.373239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8c5b7d286444676aa1e327dc4a5a9f7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:20.381863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8c5b7d286444676aa1e327dc4a5a9f7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:20.383811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8c5b7d286444676aa1e327dc4a5a9f7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:20.386231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8c5b7d286444676aa1e327dc4a5a9f7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:20.442902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1145/2266] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:21:19 2022] DELETE /identity/v3/users/a8c5b7d286444676aa1e327dc4a5a9f7 => generated 0 bytes in 579 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:20.448277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:20.448827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:20.448827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:20.479433 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:20.480969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:20.528038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:20.612968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1122/2267] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:21:20.622637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:20.624850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:20.760068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972f66b0-48f8-4401-acf1-7456335f6ab6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:20.779407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5742213-3c35-498e-9332-214a98a97fc1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:20.780530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5742213-3c35-498e-9332-214a98a97fc1 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:20.781017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5742213-3c35-498e-9332-214a98a97fc1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:20.781607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5742213-3c35-498e-9332-214a98a97fc1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:20.795626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5742213-3c35-498e-9332-214a98a97fc1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:20.797358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5742213-3c35-498e-9332-214a98a97fc1 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:20.830959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1146/2268] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 209 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:21.567143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:21.568771 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:21.584344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfcdc3c-665c-43ee-b03d-1efa817be401 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:21.590597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:21.592597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:21.593273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:21.593632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] PATH_INFO: `/v3/projects/e56b8f071c8c49f4a511578142880b4c` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:21.598283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e56b8f071c8c49f4a511578142880b4c)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:21.600395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:21.738630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e56b8f071c8c49f4a511578142880b4c', 'request_id': 'req-2d68e214-25c5-40dc-a7dc-4f78ca48a619'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:21.746283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e56b8f071c8c49f4a511578142880b4c', 'request_id': 'req-2d68e214-25c5-40dc-a7dc-4f78ca48a619'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:21.806473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e56b8f071c8c49f4a511578142880b4c', 'request_id': 'req-2d68e214-25c5-40dc-a7dc-4f78ca48a619'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:21.820309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] The token cache is being invalidate because project e56b8f071c8c49f4a511578142880b4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:21:21.820584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:21.823212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2d68e214-25c5-40dc-a7dc-4f78ca48a619 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:21.824480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1123/2269] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:21:21 2022] DELETE /identity/v3/projects/e56b8f071c8c49f4a511578142880b4c => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:21.829610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24367611-9183-4cd4-ab63-97c42defc8d4 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:21.830112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24367611-9183-4cd4-ab63-97c42defc8d4 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:21.830192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24367611-9183-4cd4-ab63-97c42defc8d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:21.855426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-24367611-9183-4cd4-ab63-97c42defc8d4 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:21.856743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-24367611-9183-4cd4-ab63-97c42defc8d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:21.862349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:21.951610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e1a7b-ff23-497c-9b3f-e9d7498d8267 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:21.951900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e1a7b-ff23-497c-9b3f-e9d7498d8267 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:21.952365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88e1a7b-ff23-497c-9b3f-e9d7498d8267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:21.958545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1147/2270] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:21.979425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d88e1a7b-ff23-497c-9b3f-e9d7498d8267 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:21.980838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d88e1a7b-ff23-497c-9b3f-e9d7498d8267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:21.990508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1124/2271] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:21:21.995283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24367611-9183-4cd4-ab63-97c42defc8d4 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:22.005844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24367611-9183-4cd4-ab63-97c42defc8d4 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:22.012252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:22.012252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.012252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.012593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.013280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:21:22.013590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:22.015597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:22.034794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1148/2272] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:21:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:21:22.035992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e1a7b-ff23-497c-9b3f-e9d7498d8267 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:22.046164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88e1a7b-ff23-497c-9b3f-e9d7498d8267 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:22.051206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:22.051863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.052143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.052418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.053115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:22.055127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:22.180408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1125/2273] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:21:22 2022] POST /identity/v3/projects => generated 369 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:21:22.181853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:22.195729 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c11df1-ce65-41c8-a997-8bea692b5cda admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:22.271354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:22.272034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.272314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.272588 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.273287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:22.275504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:22.352342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1149/2274] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:21:22 2022] POST /identity/v3/users => generated 285 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:21:22.354340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:22.371500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6898ac87-1e91-48a6-8ce1-9f336f32e273 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:22.379569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:22.380344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.380632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.380912 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.381703 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:22.383450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:22.395250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1126/2275] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:21:22 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:21:22.399421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:22.423217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baf695d-c1fa-4dca-b51b-731c7a455c9f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:22.436776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292fb698-ef72-4883-a5e6-9650b2c52ca5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:22.437500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292fb698-ef72-4883-a5e6-9650b2c52ca5 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.437784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292fb698-ef72-4883-a5e6-9650b2c52ca5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.438064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292fb698-ef72-4883-a5e6-9650b2c52ca5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.438852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292fb698-ef72-4883-a5e6-9650b2c52ca5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:22.440496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292fb698-ef72-4883-a5e6-9650b2c52ca5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:22.453461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1150/2276] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:21:22 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:21:22.456908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:22.471276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01a0829-e9d6-491a-8df5-18ba98eec6ab admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:22.478167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:22.480576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.481048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.481478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] PATH_INFO: `/v3/projects/879dcfac42504bb582a71a6b5d9c4892/users/1992931800ff45658730ac9e17eb9b35/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.499043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] RBAC: Authorizing `identity:create_grant(project_id=879dcfac42504bb582a71a6b5d9c4892, user_id=1992931800ff45658730ac9e17eb9b35, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:22.501695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:22.631678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1127/2277] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:21:22 2022] PUT /identity/v3/projects/879dcfac42504bb582a71a6b5d9c4892/users/1992931800ff45658730ac9e17eb9b35/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:22.645438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac932098-cce6-4916-9a0f-85a3fe5c43b3 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.646123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac932098-cce6-4916-9a0f-85a3fe5c43b3 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.646579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac932098-cce6-4916-9a0f-85a3fe5c43b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.707868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ac932098-cce6-4916-9a0f-85a3fe5c43b3 None None] MFA Rules not processed for user `1992931800ff45658730ac9e17eb9b35`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:22.709156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ac932098-cce6-4916-9a0f-85a3fe5c43b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:22.862150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1151/2278] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:22 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:22.883142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:22.885258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:22.958801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec6ee3e-95bc-4ddc-a097-7a7fadf3a0ea admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:22.962623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22e732b-75f0-48c1-aa57-5d8820c1eae9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:22.963328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e732b-75f0-48c1-aa57-5d8820c1eae9 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:22.963612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e732b-75f0-48c1-aa57-5d8820c1eae9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:22.963889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e732b-75f0-48c1-aa57-5d8820c1eae9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:22.970686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:22.973365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22e732b-75f0-48c1-aa57-5d8820c1eae9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:22.974937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22e732b-75f0-48c1-aa57-5d8820c1eae9 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:22.985316 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1128/2279] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:22 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:23.012298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac932098-cce6-4916-9a0f-85a3fe5c43b3 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:23.017794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:23.089168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac932098-cce6-4916-9a0f-85a3fe5c43b3 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:23.094597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:23.095490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:23.101667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:23.101667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] PATH_INFO: `/v3/users/1992931800ff45658730ac9e17eb9b35` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:23.101667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1992931800ff45658730ac9e17eb9b35)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:23.101667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:23.366250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1992931800ff45658730ac9e17eb9b35'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:23.544876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1992931800ff45658730ac9e17eb9b35'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:23.544876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1992931800ff45658730ac9e17eb9b35'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:23.544876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1992931800ff45658730ac9e17eb9b35'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:23.544876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1992931800ff45658730ac9e17eb9b35'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:23.544876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1152/2280] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:21:23 2022] DELETE /identity/v3/users/1992931800ff45658730ac9e17eb9b35 => generated 0 bytes in 525 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:23.551688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:23.552179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:23.552647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:23.575381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:23.576632 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:23.657647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1129/2281] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:23.668417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:23.670072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:23.741143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3cb99d3-447c-43d8-9f08-71dcdb274669 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:23.749445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-977d0f34-a872-464a-be33-ed2f01a7c525 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:23.751624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977d0f34-a872-464a-be33-ed2f01a7c525 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:23.752550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977d0f34-a872-464a-be33-ed2f01a7c525 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:23.753355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-977d0f34-a872-464a-be33-ed2f01a7c525 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:23.776881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977d0f34-a872-464a-be33-ed2f01a7c525 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:23.780460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-977d0f34-a872-464a-be33-ed2f01a7c525 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:23.819718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1153/2282] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:24.427020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:24.427020 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:24.444142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db8153c6-d89b-4f08-9ba7-ee4cea10cab1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:24.450980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:24.452169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:24.452461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:24.452742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] PATH_INFO: `/v3/projects/879dcfac42504bb582a71a6b5d9c4892` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:24.453962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=879dcfac42504bb582a71a6b5d9c4892)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:24.456197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:24.550190 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '879dcfac42504bb582a71a6b5d9c4892', 'request_id': 'req-2c3efabf-e3c7-4b22-994b-5115a6b748d6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:24.556001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '879dcfac42504bb582a71a6b5d9c4892', 'request_id': 'req-2c3efabf-e3c7-4b22-994b-5115a6b748d6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:24.628074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '879dcfac42504bb582a71a6b5d9c4892', 'request_id': 'req-2c3efabf-e3c7-4b22-994b-5115a6b748d6'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:24.641182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] The token cache is being invalidate because project 879dcfac42504bb582a71a6b5d9c4892 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:21:24.641472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:24.642167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-2c3efabf-e3c7-4b22-994b-5115a6b748d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:24.643895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1130/2283] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:21:24 2022] DELETE /identity/v3/projects/879dcfac42504bb582a71a6b5d9c4892 => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:24.688075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3aae6-d0dc-46b1-9f37-628444430867 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:24.688637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3aae6-d0dc-46b1-9f37-628444430867 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:24.688743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa3aae6-d0dc-46b1-9f37-628444430867 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:24.708181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9fa3aae6-d0dc-46b1-9f37-628444430867 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:24.709426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9fa3aae6-d0dc-46b1-9f37-628444430867 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:24.758550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:24.807606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1154/2284] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:24.815335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa955e-708f-465a-9443-9a9c4d5bd038 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:24.816146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa955e-708f-465a-9443-9a9c4d5bd038 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:24.816146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6fa955e-708f-465a-9443-9a9c4d5bd038 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:24.837597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e6fa955e-708f-465a-9443-9a9c4d5bd038 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:24.838968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e6fa955e-708f-465a-9443-9a9c4d5bd038 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:24.872982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3aae6-d0dc-46b1-9f37-628444430867 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:24.879022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1131/2285] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:24.885180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa3aae6-d0dc-46b1-9f37-628444430867 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:24.890410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:24.891360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:24.891629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:24.892090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:24.892801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:21:24.893170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:24.894937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:24.912129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1155/2286] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:21:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:21:24.913810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fa955e-708f-465a-9443-9a9c4d5bd038 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:24.923964 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6fa955e-708f-465a-9443-9a9c4d5bd038 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:24.928666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:24.929519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:24.929865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:24.930071 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:24.931423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:24.933565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:25.156702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1132/2287] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:21:24 2022] POST /identity/v3/projects => generated 381 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:21:25.161616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:25.175784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962ff25e-1b57-424d-b2f3-32b4368b63b9 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:25.326321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:25.327041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:25.329785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:25.330083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:25.330859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:25.332836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:25.413010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1156/2288] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:21:25 2022] POST /identity/v3/users => generated 291 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:21:25.418004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:25.434536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cfc222d-2fb5-4747-9840-e8475bffe143 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:25.452762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:25.452762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:25.452762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:25.452762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:25.453258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:25.453258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:25.467099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1133/2289] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:21:25 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:21:25.471647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:25.482701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb2708f5-7063-4651-853b-c5a2e5efa0e4 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:25.487452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fed213a-9e43-411b-92f7-de17b7ada909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:25.488135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed213a-9e43-411b-92f7-de17b7ada909 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:25.488408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed213a-9e43-411b-92f7-de17b7ada909 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:25.488691 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fed213a-9e43-411b-92f7-de17b7ada909 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:25.489427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fed213a-9e43-411b-92f7-de17b7ada909 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:25.491247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fed213a-9e43-411b-92f7-de17b7ada909 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:25.502572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1157/2290] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:21:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:21:25.506652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:25.517257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695ea07e-9068-41e1-8698-9175948c977b admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:25.521950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:25.522874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:25.523161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:25.523437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] PATH_INFO: `/v3/projects/bae52c2cdf124894a541a6b7407eb23c/users/ea504bdceb7b4a1b8109c658e133f955/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:25.539195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bae52c2cdf124894a541a6b7407eb23c, user_id=ea504bdceb7b4a1b8109c658e133f955, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:25.542677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:25.623524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1134/2291] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:21:25 2022] PUT /identity/v3/projects/bae52c2cdf124894a541a6b7407eb23c/users/ea504bdceb7b4a1b8109c658e133f955/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:25.628579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4312e-3fae-4ece-aaf9-08e83f847bca None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:25.629100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4312e-3fae-4ece-aaf9-08e83f847bca None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:25.629386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e4312e-3fae-4ece-aaf9-08e83f847bca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:25.648739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c4e4312e-3fae-4ece-aaf9-08e83f847bca None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:25.650704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c4e4312e-3fae-4ece-aaf9-08e83f847bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:25.726417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1158/2292] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:25.737365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:25.739132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:25.759141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:25.813355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93219556-0284-428a-aa53-5be76462f1a7 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:25.817802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:25.818603 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:25.818902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:25.819187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:25.831391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:25.832989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:25.844622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1135/2293] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:27.526673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0198d7a-7156-44ca-b7d3-4e5f8a2626c9 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:27.526673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0198d7a-7156-44ca-b7d3-4e5f8a2626c9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:27.526673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0198d7a-7156-44ca-b7d3-4e5f8a2626c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:27.555278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e0198d7a-7156-44ca-b7d3-4e5f8a2626c9 None None] MFA Rules not processed for user `ea504bdceb7b4a1b8109c658e133f955`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:27.557240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e0198d7a-7156-44ca-b7d3-4e5f8a2626c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:27.611475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:27.678873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1159/2294] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:27 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:27.691479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:27.693229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:27.813759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84486bab-eb73-4c9b-8505-d04d34156882 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:27.817785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:27.818537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:27.818966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:27.819258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:27.829448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:27.831124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:27.842390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1136/2295] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:27 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:30.650548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdde09e-f7aa-452e-8ca7-50a8896e8036 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:30.651179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdde09e-f7aa-452e-8ca7-50a8896e8036 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:30.651562 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdde09e-f7aa-452e-8ca7-50a8896e8036 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:30.674111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3fdde09e-f7aa-452e-8ca7-50a8896e8036 None None] MFA Rules not processed for user `ea504bdceb7b4a1b8109c658e133f955`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:30.675392 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3fdde09e-f7aa-452e-8ca7-50a8896e8036 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:30.687843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1160/2296] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:30 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:34.641881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:34.644047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:34.750602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae59e90-72ce-4745-866f-576b09bd3a25 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:34.756161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:34.756817 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:34.757165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:34.757443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:34.760007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:34.772587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:34.774563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:34.786791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1137/2297] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:34 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:40.501096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f000fa71-c422-46a6-ab1c-cc25e985546d None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:40.501574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f000fa71-c422-46a6-ab1c-cc25e985546d None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:40.501696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f000fa71-c422-46a6-ab1c-cc25e985546d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:40.521927 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f000fa71-c422-46a6-ab1c-cc25e985546d None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:40.523547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f000fa71-c422-46a6-ab1c-cc25e985546d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:40.582256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1161/2298] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:21:40 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:40.604122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:40.605709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:40.662392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:40.681114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524badc1-93c8-4878-890a-53779d13e261 service placement] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:40.686331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:40.687041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:40.687327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:40.687607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:40.697395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:40.699018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:40.710039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1138/2299] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:21:40 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:21:41.654507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc812a1a-c315-4a16-a4e2-d36b63bfbd6a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:41.654996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc812a1a-c315-4a16-a4e2-d36b63bfbd6a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:41.655103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc812a1a-c315-4a16-a4e2-d36b63bfbd6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:41.674447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-fc812a1a-c315-4a16-a4e2-d36b63bfbd6a None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:41.675703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fc812a1a-c315-4a16-a4e2-d36b63bfbd6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:41.683923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1162/2300] 10.222.0.27 () {58 vars in 967 bytes} [Thu Jul 14 12:21:41 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:41.696223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:41.706699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01129be-abc6-4e77-a3d9-d58cadc171c3 service swift] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:41.712024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:41.712672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:41.712953 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:41.713414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:41.722948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:41.724589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:41.734822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1139/2301] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:21:41 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:21:42.354580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2ea7a0-1bb9-4623-aeb0-1396ab583a22 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:42.354905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2ea7a0-1bb9-4623-aeb0-1396ab583a22 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:42.355189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2ea7a0-1bb9-4623-aeb0-1396ab583a22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:42.376052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ad2ea7a0-1bb9-4623-aeb0-1396ab583a22 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:42.377282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ad2ea7a0-1bb9-4623-aeb0-1396ab583a22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:42.385517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1163/2302] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:21:42 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:42.397871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:42.409965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d055592-e1f3-4f02-b1dc-1762c62e3039 service swift] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:42.416400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:42.417088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:42.417544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:42.417977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:42.429222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:42.431631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:42.444044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1140/2303] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:21:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:21:43.214383 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:43.217585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:43.217585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:43.244699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:43.245934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:43.254384 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1164/2304] 10.222.0.27 () {58 vars in 967 bytes} [Thu Jul 14 12:21:43 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:21:43.267720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:43.278962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555e8a80-693b-448c-b193-55e15d4059f4 service swift] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:43.284018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a24fc8b-0adc-4396-bbf4-08a00c86fc63 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:43.284673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a24fc8b-0adc-4396-bbf4-08a00c86fc63 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:43.284950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a24fc8b-0adc-4396-bbf4-08a00c86fc63 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:43.285939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a24fc8b-0adc-4396-bbf4-08a00c86fc63 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:43.295123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a24fc8b-0adc-4396-bbf4-08a00c86fc63 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:43.296755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a24fc8b-0adc-4396-bbf4-08a00c86fc63 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:43.306369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1141/2305] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:21:43 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:21:45.984601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:45.986040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:46.002477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de50a55d-67b2-473f-aac0-ab8a40a29862 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:46.009228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:46.010206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:46.010667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:46.011145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] PATH_INFO: `/v3/users/ca8ddc2748da4b05bc6d7ee16ee3db24` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:46.012413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca8ddc2748da4b05bc6d7ee16ee3db24)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:46.014491 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:46.218116 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca8ddc2748da4b05bc6d7ee16ee3db24'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.295960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca8ddc2748da4b05bc6d7ee16ee3db24'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.303625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca8ddc2748da4b05bc6d7ee16ee3db24'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.304260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca8ddc2748da4b05bc6d7ee16ee3db24'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.305452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca8ddc2748da4b05bc6d7ee16ee3db24'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.324746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1165/2306] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:21:45 2022] DELETE /identity/v3/users/ca8ddc2748da4b05bc6d7ee16ee3db24 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:46.329321 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:46.329791 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:46.329884 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:46.347516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:46.348931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:46.409470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1142/2307] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:21:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:46.419719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:46.421133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:46.438638 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:46.482510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ab25bad-d4aa-420f-992e-0033dd773e49 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:46.486139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc93307d-d70d-4d85-9a4d-a2921d5d2552 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:46.486807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc93307d-d70d-4d85-9a4d-a2921d5d2552 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:46.487095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc93307d-d70d-4d85-9a4d-a2921d5d2552 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:46.487373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc93307d-d70d-4d85-9a4d-a2921d5d2552 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:46.496533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc93307d-d70d-4d85-9a4d-a2921d5d2552 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:46.497977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc93307d-d70d-4d85-9a4d-a2921d5d2552 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:46.512134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1166/2308] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:21:46.743263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:46.744674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:46.756886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2ef510-cdff-4435-91e1-814aa33ab6a3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:46.761819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:46.772715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:46.772715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:46.772715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] PATH_INFO: `/v3/projects/206723edeb144547882daa902124ed7c` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:46.772715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] RBAC: Authorizing `identity:delete_project(project_id=206723edeb144547882daa902124ed7c)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:46.772715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:46.907299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '206723edeb144547882daa902124ed7c', 'request_id': 'req-8244ed11-e60e-4e73-b313-4c04db5db8dd'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.915083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '206723edeb144547882daa902124ed7c', 'request_id': 'req-8244ed11-e60e-4e73-b313-4c04db5db8dd'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.982705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '206723edeb144547882daa902124ed7c', 'request_id': 'req-8244ed11-e60e-4e73-b313-4c04db5db8dd'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.998224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] The token cache is being invalidate because project 206723edeb144547882daa902124ed7c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:21:46.998527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:46.999221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:21:47.001011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1143/2309] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:21:46 2022] DELETE /identity/v3/projects/206723edeb144547882daa902124ed7c => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:21:55.213114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:55.214256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:55.216708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:55.238164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:21:55.239458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:55.312924 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1167/2310] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:21:55 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:21:55.325731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:55.327273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:55.359269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:21:55.402100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8244ed11-e60e-4e73-b313-4c04db5db8dd admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:55.407651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe54ac35-403f-4cf0-843f-80b30d5c01e6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:55.408546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe54ac35-403f-4cf0-843f-80b30d5c01e6 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:55.408974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe54ac35-403f-4cf0-843f-80b30d5c01e6 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:55.409337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe54ac35-403f-4cf0-843f-80b30d5c01e6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:55.418572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe54ac35-403f-4cf0-843f-80b30d5c01e6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:55.420313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe54ac35-403f-4cf0-843f-80b30d5c01e6 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:55.428942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1144/2311] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:21:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:21:59.321631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:21:59.326649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:59.468946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24dbee32-155d-4617-8046-80a2561693a4 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:21:59.480710 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e11ce-4528-46d3-a3ac-8293042e07bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:21:59.481317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e11ce-4528-46d3-a3ac-8293042e07bc service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:21:59.481606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e11ce-4528-46d3-a3ac-8293042e07bc service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:21:59.481890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e11ce-4528-46d3-a3ac-8293042e07bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:21:59.486749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-dd8e11ce-4528-46d3-a3ac-8293042e07bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:21:59.616176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8e11ce-4528-46d3-a3ac-8293042e07bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:21:59.621284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8e11ce-4528-46d3-a3ac-8293042e07bc service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:21:59.641872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1168/2312] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:21:59 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:22:05.495441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=21 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:22:09.625847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:22:13.439639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0f920-1cad-4ea3-a9a9-8cc8e5f525f8 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:13.440203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0f920-1cad-4ea3-a9a9-8cc8e5f525f8 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:13.440203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fa0f920-1cad-4ea3-a9a9-8cc8e5f525f8 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:13.441892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1145/2313] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:22:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:22:13.449082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1c408-0cb6-4798-8126-671ecb4cdb71 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:13.449597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1c408-0cb6-4798-8126-671ecb4cdb71 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:13.449686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c1c408-0cb6-4798-8126-671ecb4cdb71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:13.468017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-32c1c408-0cb6-4798-8126-671ecb4cdb71 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:22:13.469401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-32c1c408-0cb6-4798-8126-671ecb4cdb71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:13.537600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1169/2314] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:22:13 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:22:13.549132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa0f920-1cad-4ea3-a9a9-8cc8e5f525f8 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:22:13.551128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7fa0f920-1cad-4ea3-a9a9-8cc8e5f525f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:13.608616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fa0f920-1cad-4ea3-a9a9-8cc8e5f525f8 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:22:13.612102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba863d4f-fa1b-4190-94f0-d49cea0e840a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:22:13.612746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba863d4f-fa1b-4190-94f0-d49cea0e840a service glance] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:13.613028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba863d4f-fa1b-4190-94f0-d49cea0e840a service glance] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:13.613306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba863d4f-fa1b-4190-94f0-d49cea0e840a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:13.624014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba863d4f-fa1b-4190-94f0-d49cea0e840a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:22:13.625449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba863d4f-fa1b-4190-94f0-d49cea0e840a service glance] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:22:13.636807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1146/2315] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:22:13 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:22:13.833122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043e692-871f-4c16-8391-8c31dfa4025c None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:13.833417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043e692-871f-4c16-8391-8c31dfa4025c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:13.833701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043e692-871f-4c16-8391-8c31dfa4025c None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:13.835139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1170/2316] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:22:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:22:13.839928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da5bf99-d52c-466a-bd12-16c90dfe2a24 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:13.840771 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da5bf99-d52c-466a-bd12-16c90dfe2a24 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:13.840771 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da5bf99-d52c-466a-bd12-16c90dfe2a24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:13.859447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4da5bf99-d52c-466a-bd12-16c90dfe2a24 None None] MFA Rules not processed for user `a4fff90ae5434b58932f468ca9c827f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:22:13.860586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4da5bf99-d52c-466a-bd12-16c90dfe2a24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:13.867172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1147/2317] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:22:13 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:22:13.879535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043e692-871f-4c16-8391-8c31dfa4025c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:22:13.881106 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5043e692-871f-4c16-8391-8c31dfa4025c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:13.894474 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043e692-871f-4c16-8391-8c31dfa4025c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:22:13.897959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e736671-07e4-4dac-92b1-e87283f02265 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:22:13.898859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e736671-07e4-4dac-92b1-e87283f02265 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:13.899226 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e736671-07e4-4dac-92b1-e87283f02265 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:13.899502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e736671-07e4-4dac-92b1-e87283f02265 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:13.906379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e736671-07e4-4dac-92b1-e87283f02265 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:22:13.907850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e736671-07e4-4dac-92b1-e87283f02265 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:22:13.916162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1171/2318] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:22:13 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:22:14.000869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631fd83-2487-4a61-a2c7-55e0d8704132 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:14.001159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631fd83-2487-4a61-a2c7-55e0d8704132 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:14.001440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a631fd83-2487-4a61-a2c7-55e0d8704132 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:14.002911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1148/2319] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:22:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:22:14.008935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7d23f7-b894-4fe7-a18d-0076ea181f01 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:14.009238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7d23f7-b894-4fe7-a18d-0076ea181f01 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:14.009563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7d23f7-b894-4fe7-a18d-0076ea181f01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:14.028722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4d7d23f7-b894-4fe7-a18d-0076ea181f01 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:22:14.030045 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4d7d23f7-b894-4fe7-a18d-0076ea181f01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:14.039647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1172/2320] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:22:14 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:22:14.050165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a631fd83-2487-4a61-a2c7-55e0d8704132 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:22:14.061403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a631fd83-2487-4a61-a2c7-55e0d8704132 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:22:14.065114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:22:14.065751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:14.066055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:14.066337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:14.077524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:22:14.078986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:22:14.090694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1149/2321] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:22:14 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:22:24.023155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=40 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:22:24.084985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=35 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:22:29.251888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f20e4-0b30-4fe0-bd7d-71c4763c6820 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:29.252359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f20e4-0b30-4fe0-bd7d-71c4763c6820 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:29.252452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4f20e4-0b30-4fe0-bd7d-71c4763c6820 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:29.271082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-fd4f20e4-0b30-4fe0-bd7d-71c4763c6820 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:22:29.272256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fd4f20e4-0b30-4fe0-bd7d-71c4763c6820 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:29.279597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1173/2322] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:22:29 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:22:29.291307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:22:29.302734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71abf557-9b3c-458a-893b-a75f9eb0d75f service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:22:29.308495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83e9d7ca-a320-4148-85e1-c58806e44122 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:22:29.309138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9d7ca-a320-4148-85e1-c58806e44122 service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:29.309427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9d7ca-a320-4148-85e1-c58806e44122 service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:29.309706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83e9d7ca-a320-4148-85e1-c58806e44122 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:29.317948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e9d7ca-a320-4148-85e1-c58806e44122 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:22:29.319622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83e9d7ca-a320-4148-85e1-c58806e44122 service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:22:29.328475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1150/2323] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:22:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:22:39.265092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:22:39.325125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=6 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:22:57.155142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9c55c1-55c9-41b2-bef5-45bce351cc10 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:57.155730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9c55c1-55c9-41b2-bef5-45bce351cc10 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:57.155730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9c55c1-55c9-41b2-bef5-45bce351cc10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:57.173622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-7b9c55c1-55c9-41b2-bef5-45bce351cc10 None None] MFA Rules not processed for user `ea504bdceb7b4a1b8109c658e133f955`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:22:57.174946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7b9c55c1-55c9-41b2-bef5-45bce351cc10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:57.230056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1174/2324] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:22:57 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:22:57.295628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9473837a-0e93-4b2e-8fac-58c4d847c567 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:57.296210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9473837a-0e93-4b2e-8fac-58c4d847c567 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:57.296346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9473837a-0e93-4b2e-8fac-58c4d847c567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:57.318727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9473837a-0e93-4b2e-8fac-58c4d847c567 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:22:57.320012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9473837a-0e93-4b2e-8fac-58c4d847c567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:22:57.327420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1151/2325] 10.222.0.27 () {58 vars in 967 bytes} [Thu Jul 14 12:22:57 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:22:57.350363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9c55c1-55c9-41b2-bef5-45bce351cc10 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:22:57.362006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9c55c1-55c9-41b2-bef5-45bce351cc10 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:22:57.367715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:22:57.368360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:22:57.368642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:22:57.368914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:22:57.377937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:22:57.379657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:22:57.388130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1175/2326] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:22:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:23:07.310861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:23:07.384468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=31 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:23:14.796588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350deef0-55d4-46d8-adf4-386a15969436 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:14.797095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350deef0-55d4-46d8-adf4-386a15969436 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:14.797222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350deef0-55d4-46d8-adf4-386a15969436 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:14.815293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-350deef0-55d4-46d8-adf4-386a15969436 None None] MFA Rules not processed for user `d6da4b0616034aa1bb22c44a8f5ef280`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:23:14.816612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-350deef0-55d4-46d8-adf4-386a15969436 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:14.872941 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1152/2327] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:23:14 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:23:17.805273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:17.814911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b366718e-611d-4faf-93fb-362c43bb6bab service swift] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:17.819925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:17.820574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:17.820857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:17.821137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:17.822950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:17.836966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:17.838402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:17.851589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1176/2328] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:17 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:18.129703 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350deef0-55d4-46d8-adf4-386a15969436 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:18.139714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350deef0-55d4-46d8-adf4-386a15969436 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:18.143193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:18.143817 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:18.144105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:18.144401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:18.146246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:18.161456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:18.163055 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:18.174904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1153/2329] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:18 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:21.325498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:21.326946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:21.340794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e91626-5364-4c9e-8ae0-a9fb9026a1c7 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:21.346149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dba468d-22d4-4363-bb4f-0ccfdf4940a0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:21.346798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dba468d-22d4-4363-bb4f-0ccfdf4940a0 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:21.347081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dba468d-22d4-4363-bb4f-0ccfdf4940a0 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:21.347354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dba468d-22d4-4363-bb4f-0ccfdf4940a0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:21.349156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7dba468d-22d4-4363-bb4f-0ccfdf4940a0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:21.359101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dba468d-22d4-4363-bb4f-0ccfdf4940a0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:21.360668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dba468d-22d4-4363-bb4f-0ccfdf4940a0 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:21.370001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1177/2330] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:21 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:28.169284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=31 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:23:31.127978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:31.136527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d80d26-a295-42f2-9225-378fc620d843 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:31.140350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:31.140984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:31.141265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:31.141536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:31.143482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:31.211415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:31.212945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:31.223202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1154/2331] 10.222.0.27 () {60 vars in 1396 bytes} [Thu Jul 14 12:23:31 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3260 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:31.352667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:31.353165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:31.353266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:31.370187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] MFA Rules not processed for user `0d302e57946c42d799e9bf120873eba6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:23:31.371421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:31.378323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1178/2332] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:23:31 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:23:41.218045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=28 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:23:41.365164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=15 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:23:50.093681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:50.104377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeddd96d-ac0b-43e1-8839-fe0fda6ab337 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:50.108326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00476c22-8931-4c30-9c31-bb04215d200d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:50.109090 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00476c22-8931-4c30-9c31-bb04215d200d service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:50.109418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00476c22-8931-4c30-9c31-bb04215d200d service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:50.109874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00476c22-8931-4c30-9c31-bb04215d200d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:50.111932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-00476c22-8931-4c30-9c31-bb04215d200d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:50.123523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00476c22-8931-4c30-9c31-bb04215d200d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:50.125081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00476c22-8931-4c30-9c31-bb04215d200d service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:50.135834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1155/2333] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:50 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:55.651617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:55.653122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:55.678559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59f00a7-dd5b-4b84-8129-15de0942c0d3 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:55.684304 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bfd307-9355-4f76-8710-d934e21101cd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:55.685343 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bfd307-9355-4f76-8710-d934e21101cd service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:55.685786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bfd307-9355-4f76-8710-d934e21101cd service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:55.686216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bfd307-9355-4f76-8710-d934e21101cd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:55.688405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-54bfd307-9355-4f76-8710-d934e21101cd service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:55.703841 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bfd307-9355-4f76-8710-d934e21101cd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:55.705267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bfd307-9355-4f76-8710-d934e21101cd service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:55.732637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1179/2334] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:55 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:58.562416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6ec8d8-b51f-44c0-a8af-e8c36afbb906 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:58.563056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6ec8d8-b51f-44c0-a8af-e8c36afbb906 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:58.563056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6ec8d8-b51f-44c0-a8af-e8c36afbb906 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:58.564541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1156/2335] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:23:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:23:58.570517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b773fb-8418-45bf-81c7-f2349e77a12e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:58.571081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b773fb-8418-45bf-81c7-f2349e77a12e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:58.571081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21b773fb-8418-45bf-81c7-f2349e77a12e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:58.590150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-21b773fb-8418-45bf-81c7-f2349e77a12e None None] MFA Rules not processed for user `a4fff90ae5434b58932f468ca9c827f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:23:58.591397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-21b773fb-8418-45bf-81c7-f2349e77a12e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:58.598694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1180/2336] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:23:58 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:23:58.610663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6ec8d8-b51f-44c0-a8af-e8c36afbb906 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:58.622017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6ec8d8-b51f-44c0-a8af-e8c36afbb906 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:58.625570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c2665c-ff6c-4e07-b11a-485bc1e77e4b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:58.626202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2665c-ff6c-4e07-b11a-485bc1e77e4b service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:58.626486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2665c-ff6c-4e07-b11a-485bc1e77e4b service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:58.626784 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c2665c-ff6c-4e07-b11a-485bc1e77e4b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:58.633925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c2665c-ff6c-4e07-b11a-485bc1e77e4b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:58.635396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97c2665c-ff6c-4e07-b11a-485bc1e77e4b service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:58.643353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1157/2337] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:58 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:58.728429 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d289bf2-8c92-4b7d-9820-a6cbed1651cd None None] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:58.728719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d289bf2-8c92-4b7d-9820-a6cbed1651cd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:58.729004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d289bf2-8c92-4b7d-9820-a6cbed1651cd None None] PATH_INFO: `/` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:58.730391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1181/2338] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:23:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:23:58.735434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:58.735724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:58.736000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:58.755109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:23:58.756343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:58.765569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1158/2339] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:23:58 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:23:58.776255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d289bf2-8c92-4b7d-9820-a6cbed1651cd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:58.788391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d289bf2-8c92-4b7d-9820-a6cbed1651cd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:58.791982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f567b5c-d572-47b2-9ed9-92055ecb505b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:58.792615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f567b5c-d572-47b2-9ed9-92055ecb505b service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:58.792897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f567b5c-d572-47b2-9ed9-92055ecb505b service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:58.793173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f567b5c-d572-47b2-9ed9-92055ecb505b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:58.804091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f567b5c-d572-47b2-9ed9-92055ecb505b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:58.805551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f567b5c-d572-47b2-9ed9-92055ecb505b service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:58.818283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1182/2340] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:58 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:23:58.885324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:23:58.886706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:58.974016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eabfdf9c-2108-41c2-b6e0-8978a3e68c6b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:23:58.979706 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:23:58.980525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:23:58.981032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:23:58.981415 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:23:58.983567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:23:58.996247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:23:58.998050 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:23:59.009856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1159/2341] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:23:58 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:03.622150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f781b1b-6536-4372-b944-44254be81f2b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:03.622660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f781b1b-6536-4372-b944-44254be81f2b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:03.622828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f781b1b-6536-4372-b944-44254be81f2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:03.641118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2f781b1b-6536-4372-b944-44254be81f2b None None] MFA Rules not processed for user `15e708082e294dd9a0834ef21ea46fd9`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:03.642316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2f781b1b-6536-4372-b944-44254be81f2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:03.699261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1183/2342] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:03 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:24:04.012295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:04.021621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ee22c8-17b4-44ae-a73f-8c8329057805 service placement] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:04.025151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:04.025820 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:04.026098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:04.026372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:04.028247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:04.038428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:04.039955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:04.048168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1160/2343] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:04 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:10.413572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f781b1b-6536-4372-b944-44254be81f2b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:10.426101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f781b1b-6536-4372-b944-44254be81f2b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:10.431469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:10.432148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:10.432432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:10.432706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:10.434513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:10.448595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:10.450404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:10.460904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1184/2344] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:10 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:10.535728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:10.537310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:10.603208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c88e8e5-60c3-417b-b8f8-ac7f10919a32 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:10.607722 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:10.608589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:10.608876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:10.609154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] PATH_INFO: `/v3/users/ea504bdceb7b4a1b8109c658e133f955` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:10.610356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea504bdceb7b4a1b8109c658e133f955)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:10.613223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:10.787860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:10.789396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea504bdceb7b4a1b8109c658e133f955'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:10.877756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea504bdceb7b4a1b8109c658e133f955'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:10.878405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea504bdceb7b4a1b8109c658e133f955'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:10.886119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea504bdceb7b4a1b8109c658e133f955'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:10.886759 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-544d4fa4-f17f-4ed5-9a37-f4081b3bd1b5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea504bdceb7b4a1b8109c658e133f955'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:10.904666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1161/2345] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:24:10 2022] DELETE /identity/v3/users/ea504bdceb7b4a1b8109c658e133f955 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:11.142466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:11.154606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed40bc6-451b-40bd-9422-a7a66d0ef243 service placement] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:11.216757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:11.217616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.217896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.218196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] PATH_INFO: `/v3/projects/bae52c2cdf124894a541a6b7407eb23c` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.219400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bae52c2cdf124894a541a6b7407eb23c)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:11.225984 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:11.322825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bae52c2cdf124894a541a6b7407eb23c', 'request_id': 'req-19e19a19-dc78-461e-8044-e5ad850616a3'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:11.386081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bae52c2cdf124894a541a6b7407eb23c', 'request_id': 'req-19e19a19-dc78-461e-8044-e5ad850616a3'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:11.393347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bae52c2cdf124894a541a6b7407eb23c', 'request_id': 'req-19e19a19-dc78-461e-8044-e5ad850616a3'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:11.394632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:11.404801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] The token cache is being invalidate because project bae52c2cdf124894a541a6b7407eb23c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:24:11.405095 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:11.405777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-19e19a19-dc78-461e-8044-e5ad850616a3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:11.407466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1185/2346] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:24:11 2022] DELETE /identity/v3/projects/bae52c2cdf124894a541a6b7407eb23c => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:11.413303 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326be4ac-43bc-439f-a9de-a96b3cccd66d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.413620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326be4ac-43bc-439f-a9de-a96b3cccd66d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.415175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-326be4ac-43bc-439f-a9de-a96b3cccd66d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.432933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-326be4ac-43bc-439f-a9de-a96b3cccd66d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:11.439104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-326be4ac-43bc-439f-a9de-a96b3cccd66d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:11.498869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1162/2347] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:11.507096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df0177d-07d1-475e-b4e1-a4dea7053069 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.507641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df0177d-07d1-475e-b4e1-a4dea7053069 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.507740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df0177d-07d1-475e-b4e1-a4dea7053069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.526545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4df0177d-07d1-475e-b4e1-a4dea7053069 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:11.527797 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4df0177d-07d1-475e-b4e1-a4dea7053069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:11.536829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1186/2348] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:11.541065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326be4ac-43bc-439f-a9de-a96b3cccd66d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:11.551501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-326be4ac-43bc-439f-a9de-a96b3cccd66d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:11.556821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:11.557497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.557781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.558061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.559045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:24:11.559360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:11.560960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:11.574682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1163/2349] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:24:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:24:11.579003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df0177d-07d1-475e-b4e1-a4dea7053069 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:11.589023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df0177d-07d1-475e-b4e1-a4dea7053069 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:11.593702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:11.594409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.594693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.594997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.595895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:11.597786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:11.696781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1187/2350] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:24:11 2022] POST /identity/v3/projects => generated 399 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:24:11.701104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:11.712385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-172414f5-ec92-4a67-8222-43b00bec1f12 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:11.770740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:11.771432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.771714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.771991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.772684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:11.774648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:11.895335 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1164/2351] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:24:11 2022] POST /identity/v3/users => generated 300 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:11.899370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:11.910070 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3256d2-802d-4559-9795-c34ec02dfd15 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:11.914681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:11.915373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.915654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.915926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.916652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:11.918316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:11.930028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1188/2352] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:11.934115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:11.944796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f785f4a-69a0-493b-b40e-6b88a52ed8ec admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:11.949417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2e136f-67d5-43b5-ab7e-3039d56ac63f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:11.950076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2e136f-67d5-43b5-ab7e-3039d56ac63f admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.950387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2e136f-67d5-43b5-ab7e-3039d56ac63f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.950669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2e136f-67d5-43b5-ab7e-3039d56ac63f admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.951432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2e136f-67d5-43b5-ab7e-3039d56ac63f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:11.953033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2e136f-67d5-43b5-ab7e-3039d56ac63f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:11.963108 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1165/2353] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:11.967111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:11.977190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ea573-d1ff-447d-9314-9dc9a23378b4 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:11.981687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:11.982559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:11.982870 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:11.983146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] PATH_INFO: `/v3/projects/c52b0c8bc2b8475cbc8ebccc81b1d28e/users/c2a73a76650b4537b377347520f09a77/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:11.998266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c52b0c8bc2b8475cbc8ebccc81b1d28e, user_id=c2a73a76650b4537b377347520f09a77, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:12.000941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:12.122921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1189/2354] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:24:11 2022] PUT /identity/v3/projects/c52b0c8bc2b8475cbc8ebccc81b1d28e/users/c2a73a76650b4537b377347520f09a77/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:12.133425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:12.133710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:12.133987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:12.163114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] MFA Rules not processed for user `c2a73a76650b4537b377347520f09a77`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:12.163114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:12.178412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=93,keystone:INSERT=4,keystone:DELETE=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:12.229049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1166/2355] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:12 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:12.241110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:12.243187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:12.322242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26078b00-2e91-4dc1-946a-9f0bbf5d856c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:12.326006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:12.326657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:12.326973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:12.327246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:12.336227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:12.337658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:12.347488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1190/2356] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:12 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:14.037711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:14.039091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:14.143680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76204dc1-88b0-4d65-9109-1632c9fe70be None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:14.151477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:14.152263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:14.152552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:14.152826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:14.154677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:14.168515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:14.170403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:14.196193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1167/2357] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:14 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 159 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:14.439192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:14.441525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:14.461463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa045cf-aac1-48cd-b82c-041d0f275348 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:14.467359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfe9277-804f-4902-bb0d-a31f88cb1200 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:14.468008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe9277-804f-4902-bb0d-a31f88cb1200 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:14.468292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe9277-804f-4902-bb0d-a31f88cb1200 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:14.468699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfe9277-804f-4902-bb0d-a31f88cb1200 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:14.477396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-abfe9277-804f-4902-bb0d-a31f88cb1200 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:14.497816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe9277-804f-4902-bb0d-a31f88cb1200 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:14.499590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfe9277-804f-4902-bb0d-a31f88cb1200 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:14.513240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1191/2358] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:14 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:21.351477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:21.414429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159edff7-288c-4e0c-966b-c55434f63434 service nova] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:21.419384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:21.420254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:21.420540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:21.420816 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] PATH_INFO: `/v3/users/c2a73a76650b4537b377347520f09a77` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:21.422043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] RBAC: Authorizing `identity:delete_user(user_id=c2a73a76650b4537b377347520f09a77)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:21.424656 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:21.553018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:21.554607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c2a73a76650b4537b377347520f09a77'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:21.620114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c2a73a76650b4537b377347520f09a77'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:21.620773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c2a73a76650b4537b377347520f09a77'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:21.632367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c2a73a76650b4537b377347520f09a77'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:21.632367 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c2a73a76650b4537b377347520f09a77'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:21.648903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1168/2359] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:24:21 2022] DELETE /identity/v3/users/c2a73a76650b4537b377347520f09a77 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:21.653452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:21.653915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:21.654047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:21.672658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:21.673882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:21.739443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1192/2360] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:24:21.749795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:21.751498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:21.813596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8bda14-ab59-4c89-8e44-8406a6b607fa admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:21.817901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e0b2af3-0b9a-4b43-9eba-b5f13648854c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:21.818551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b2af3-0b9a-4b43-9eba-b5f13648854c service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:21.818881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b2af3-0b9a-4b43-9eba-b5f13648854c service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:21.819142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e0b2af3-0b9a-4b43-9eba-b5f13648854c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:21.830389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0b2af3-0b9a-4b43-9eba-b5f13648854c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:21.831952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e0b2af3-0b9a-4b43-9eba-b5f13648854c service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:21.841549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1169/2361] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:21 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:22.377143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:22.378597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:22.393421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f24005d-7a6a-4cda-a5ac-7df3ea7a8221 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:22.398511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:22.400001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:22.400466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:22.401255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] PATH_INFO: `/v3/projects/c52b0c8bc2b8475cbc8ebccc81b1d28e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:22.402405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c52b0c8bc2b8475cbc8ebccc81b1d28e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:22.404485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:22.415896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=2,keystone:SELECT=96,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:22.540356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c52b0c8bc2b8475cbc8ebccc81b1d28e', 'request_id': 'req-e487682a-bbae-4194-b2c9-4eb3539e02f4'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:22.614373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c52b0c8bc2b8475cbc8ebccc81b1d28e', 'request_id': 'req-e487682a-bbae-4194-b2c9-4eb3539e02f4'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:22.621146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c52b0c8bc2b8475cbc8ebccc81b1d28e', 'request_id': 'req-e487682a-bbae-4194-b2c9-4eb3539e02f4'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:22.632365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] The token cache is being invalidate because project c52b0c8bc2b8475cbc8ebccc81b1d28e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:24:22.632655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:22.633279 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e487682a-bbae-4194-b2c9-4eb3539e02f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:22.634843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1193/2362] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:24:22 2022] DELETE /identity/v3/projects/c52b0c8bc2b8475cbc8ebccc81b1d28e => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:22.639901 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889bf725-aae6-4ec9-be9e-8e132f8d1e45 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:22.640407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889bf725-aae6-4ec9-be9e-8e132f8d1e45 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:22.640514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-889bf725-aae6-4ec9-be9e-8e132f8d1e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:22.658325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-889bf725-aae6-4ec9-be9e-8e132f8d1e45 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:22.661008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-889bf725-aae6-4ec9-be9e-8e132f8d1e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:22.719260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1170/2363] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:22.727522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad69075b-4907-45c7-a87e-575892c1d201 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:22.727967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad69075b-4907-45c7-a87e-575892c1d201 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:22.728167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad69075b-4907-45c7-a87e-575892c1d201 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:22.748256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ad69075b-4907-45c7-a87e-575892c1d201 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:22.749453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ad69075b-4907-45c7-a87e-575892c1d201 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:22.758499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1194/2364] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:24:22.762821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889bf725-aae6-4ec9-be9e-8e132f8d1e45 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:22.773434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-889bf725-aae6-4ec9-be9e-8e132f8d1e45 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:22.778025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:22.778685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:22.779091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:22.779378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:22.780118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:24:22.780426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:22.782019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:22.799605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1171/2365] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:24:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:24:22.800780 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad69075b-4907-45c7-a87e-575892c1d201 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:22.810356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad69075b-4907-45c7-a87e-575892c1d201 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:22.815950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:22.816826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:22.817365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:22.817817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:22.818813 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:22.821591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:22.999074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1195/2366] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:24:22 2022] POST /identity/v3/projects => generated 379 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:23.003215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:23.013458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c3c55fa-5bf4-4776-a4b8-d9a0f145c11c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:23.069725 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:23.079159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:23.079870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:23.080156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:23.080436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:23.081187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:23.083167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:23.179744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1172/2367] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:24:23 2022] POST /identity/v3/users => generated 290 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:23.184163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:23.195145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e7e3c1-e765-4038-b15d-60e1a8985799 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:23.200584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:23.201301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:23.201898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:23.202188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:23.202965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:23.204599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:23.218253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1196/2368] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:23 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:23.222593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:23.233137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4353f27-d575-45e4-b1db-de8f16d17141 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:23.239523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d253bf-3ad1-46cb-9276-0d1bbf476cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:23.240257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d253bf-3ad1-46cb-9276-0d1bbf476cd3 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:23.240541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d253bf-3ad1-46cb-9276-0d1bbf476cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:23.240882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d253bf-3ad1-46cb-9276-0d1bbf476cd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:23.241608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d253bf-3ad1-46cb-9276-0d1bbf476cd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:23.243283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d253bf-3ad1-46cb-9276-0d1bbf476cd3 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:23.255944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1173/2369] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:23 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:23.261265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:23.276477 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01bf7008-24e9-4cf2-8f86-407e124dcfe3 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:23.281659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:23.283081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:23.283542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:23.283973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] PATH_INFO: `/v3/projects/c0d098ca721b409385a08b91d861b295/users/d0adaa092d944202b3279d10e0f94a91/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:23.312844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0d098ca721b409385a08b91d861b295, user_id=d0adaa092d944202b3279d10e0f94a91, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:23.315762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:23.403961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1197/2370] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:24:23 2022] PUT /identity/v3/projects/c0d098ca721b409385a08b91d861b295/users/d0adaa092d944202b3279d10e0f94a91/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:23.409036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dff564-7a40-43ae-95fa-db01cee36f8e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:23.410420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dff564-7a40-43ae-95fa-db01cee36f8e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:23.410993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dff564-7a40-43ae-95fa-db01cee36f8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:23.444242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-70dff564-7a40-43ae-95fa-db01cee36f8e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:23.445482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-70dff564-7a40-43ae-95fa-db01cee36f8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:23.524878 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1174/2371] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:23.547370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:23.549146 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:23.637967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5555a227-d9b5-4ad2-bbad-1e05e277cb1b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:23.641821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:23.642547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:23.642939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:23.643315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:23.654349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:23.664072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:23.665525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1198/2372] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:31.324596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901e90dc-62ef-421b-b018-9a9f6f037de1 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:31.325151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901e90dc-62ef-421b-b018-9a9f6f037de1 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:31.325151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-901e90dc-62ef-421b-b018-9a9f6f037de1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:31.359945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-901e90dc-62ef-421b-b018-9a9f6f037de1 None None] MFA Rules not processed for user `d0adaa092d944202b3279d10e0f94a91`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:31.361162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-901e90dc-62ef-421b-b018-9a9f6f037de1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:31.436708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1175/2373] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:31 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:31.441554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:31.451565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af66a1d-ccb7-4040-b813-d36b628639e9 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:31.460200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:31.460882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:31.461167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:31.461441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:31.462173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:31.464105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:31.554447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1199/2374] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:24:31 2022] POST /identity/v3/projects => generated 377 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:31.558691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901e90dc-62ef-421b-b018-9a9f6f037de1 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:31.570556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-901e90dc-62ef-421b-b018-9a9f6f037de1 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:31.639590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:31.640465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:31.640848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:31.641424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:31.642294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:31.649193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:31.653639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:31.902289 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1176/2375] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:24:31 2022] POST /identity/v3/users => generated 295 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:31.903470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:31.917142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf82cc70-9bd3-4d32-b64b-1f931caf4eea admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:31.930193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:31.930941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:31.931224 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:31.931506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:31.932241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:31.934028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:31.949953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1200/2376] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:31 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:31.951146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:31.964082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b44a1b-0cbc-428d-bbe0-ba646820ed32 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:31.981359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:31.982550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:31.983999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:31.984486 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] PATH_INFO: `/v3/projects/7c864497be59453182efbe81fdf948cb/users/1738f47382234337901606d8ae00e910/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:32.018639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c864497be59453182efbe81fdf948cb, user_id=1738f47382234337901606d8ae00e910, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:32.021400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:32.199717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1177/2377] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:24:31 2022] PUT /identity/v3/projects/7c864497be59453182efbe81fdf948cb/users/1738f47382234337901606d8ae00e910/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:24:32.203402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:32.210447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:32.215418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a0a8bda-f3c3-4dd1-98e9-2170bb1f4449 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:32.317441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aedb3a8-102f-4b35-9ad9-1d615d2a6f08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:32.318129 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aedb3a8-102f-4b35-9ad9-1d615d2a6f08 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:32.318419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aedb3a8-102f-4b35-9ad9-1d615d2a6f08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:32.318696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aedb3a8-102f-4b35-9ad9-1d615d2a6f08 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:32.322988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aedb3a8-102f-4b35-9ad9-1d615d2a6f08 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:32.325004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aedb3a8-102f-4b35-9ad9-1d615d2a6f08 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:32.360642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:32.363820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1201/2378] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:32 2022] GET /identity/v3/roles => generated 1420 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:32.380341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884d07e0-9c49-4553-beca-546e686387df admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:32.395130 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:32.398011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:32.398279 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:32.404014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] PATH_INFO: `/v3/projects/7c864497be59453182efbe81fdf948cb/users/1738f47382234337901606d8ae00e910/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:32.413345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] RBAC: Authorizing `identity:create_grant(project_id=7c864497be59453182efbe81fdf948cb, user_id=1738f47382234337901606d8ae00e910, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:32.416502 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:32.431170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1178/2379] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:24:32 2022] PUT /identity/v3/projects/7c864497be59453182efbe81fdf948cb/users/1738f47382234337901606d8ae00e910/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 73 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:33.543327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8356ea8-e202-4e14-b4f6-3328f64bb9d0 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:33.544442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8356ea8-e202-4e14-b4f6-3328f64bb9d0 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:33.544916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8356ea8-e202-4e14-b4f6-3328f64bb9d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:33.574079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b8356ea8-e202-4e14-b4f6-3328f64bb9d0 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:33.575674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b8356ea8-e202-4e14-b4f6-3328f64bb9d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:33.686948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1202/2380] 10.222.0.27 () {58 vars in 967 bytes} [Thu Jul 14 12:24:33 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:33.897863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:33.900851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:34.004497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20916e3-8783-474b-8c1a-284c9baf137e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:34.010883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:34.011601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:34.012041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:34.012441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:34.025915 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:34.027630 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:34.042545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1179/2381] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:24:33 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:24:40.324456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:40.326069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:40.326591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:40.358444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] MFA Rules not processed for user `1738f47382234337901606d8ae00e910`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:40.360033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:40.576888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1203/2382] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:40 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:24:40.592988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:40.596254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:40.734094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9d891cf-25cb-4d2e-90bd-94b9783efc5d service swift] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:40.744903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:40.744903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:40.744903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:40.744903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:40.817424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:40.839845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:40.842276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:40.865613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1180/2383] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:40 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 271 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:45.897829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:45.900180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:45.938558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:45.975979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db776c2c-c2e0-4e97-8b55-0c9591376742 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:45.984804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:45.986542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:45.986883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:45.987866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] PATH_INFO: `/v3/users/c72a81dcca3145fe9f9623e49baba829` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:45.989336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=c72a81dcca3145fe9f9623e49baba829)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:45.991309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:46.207620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c72a81dcca3145fe9f9623e49baba829'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.287897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c72a81dcca3145fe9f9623e49baba829'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.315489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c72a81dcca3145fe9f9623e49baba829'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.316254 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c72a81dcca3145fe9f9623e49baba829'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.317046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c72a81dcca3145fe9f9623e49baba829'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.339370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1204/2384] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:24:45 2022] DELETE /identity/v3/users/c72a81dcca3145fe9f9623e49baba829 => generated 0 bytes in 442 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:46.598599 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:46.614251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d317f5e4-defb-454b-9541-13a170c6a66b service nova] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:46.683047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:46.684283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:46.684566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:46.684833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] PATH_INFO: `/v3/projects/a37bbdf2bd2f4392ab2ddb60d6b6ad7c` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:46.686053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] RBAC: Authorizing `identity:delete_project(project_id=a37bbdf2bd2f4392ab2ddb60d6b6ad7c)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:46.688049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:46.804666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a37bbdf2bd2f4392ab2ddb60d6b6ad7c', 'request_id': 'req-d43cc010-8983-4cc3-afb2-b9a64316c13d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.810434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a37bbdf2bd2f4392ab2ddb60d6b6ad7c', 'request_id': 'req-d43cc010-8983-4cc3-afb2-b9a64316c13d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.866080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a37bbdf2bd2f4392ab2ddb60d6b6ad7c', 'request_id': 'req-d43cc010-8983-4cc3-afb2-b9a64316c13d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.885868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] The token cache is being invalidate because project a37bbdf2bd2f4392ab2ddb60d6b6ad7c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:24:46.886180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.886856 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:46.888942 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1181/2385] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:24:46 2022] DELETE /identity/v3/projects/a37bbdf2bd2f4392ab2ddb60d6b6ad7c => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:46.893645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:46.895868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:46.910614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a88006c0-327f-49ac-8aed-bf629e1658bf admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:46.984793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:46.985887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:46.986693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:46.987001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] PATH_INFO: `/v3/users/1e473e1674374ff583113b2c38fc7752` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:46.988299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=1e473e1674374ff583113b2c38fc7752)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:46.990439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:47.209889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1e473e1674374ff583113b2c38fc7752'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:47.213882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:47.283450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1e473e1674374ff583113b2c38fc7752'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:47.291821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1e473e1674374ff583113b2c38fc7752'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:47.292926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1e473e1674374ff583113b2c38fc7752'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:47.294140 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1e473e1674374ff583113b2c38fc7752'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:47.316037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1205/2386] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:24:46 2022] DELETE /identity/v3/users/1e473e1674374ff583113b2c38fc7752 => generated 0 bytes in 423 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:47.649826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:47.651414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:47.680529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43cc010-8983-4cc3-afb2-b9a64316c13d admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:47.740989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:47.741852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:47.744990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:47.744990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] PATH_INFO: `/v3/projects/08c5082fed204967b2dd11bfbbfdce26` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:47.744990 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08c5082fed204967b2dd11bfbbfdce26)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:47.745498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:47.758864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:47.912837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08c5082fed204967b2dd11bfbbfdce26', 'request_id': 'req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:47.919142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08c5082fed204967b2dd11bfbbfdce26', 'request_id': 'req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:48.001372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08c5082fed204967b2dd11bfbbfdce26', 'request_id': 'req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:48.013993 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] The token cache is being invalidate because project 08c5082fed204967b2dd11bfbbfdce26 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:24:48.014405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:48.015091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:48.016719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1182/2387] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:24:47 2022] DELETE /identity/v3/projects/08c5082fed204967b2dd11bfbbfdce26 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:48.531700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:48.533403 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:48.595139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39df456-a0e8-448b-aae0-c8f9385b3e2c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:48.642405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:48.643088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:48.643369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:48.643646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:48.645443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:48.709417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:48.710898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:48.720883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1206/2388] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:52.360183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:52.361685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:52.433134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a534d0a-d3a7-4568-a9ca-8a3345fcd705 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:52.437969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:52.438613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:52.438947 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:52.439226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:52.441119 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:52.454681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:52.456833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:52.466969 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1183/2389] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:52 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:52.489881 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:52.491387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:52.503961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2bb513-ba87-4a75-b304-bb757d793916 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:52.508773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:52.509652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:52.509933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:52.510217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] PATH_INFO: `/v3/users/0d302e57946c42d799e9bf120873eba6` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:52.511449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d302e57946c42d799e9bf120873eba6)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:52.516580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:52.683668 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:52.697758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d302e57946c42d799e9bf120873eba6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:52.778929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d302e57946c42d799e9bf120873eba6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:52.786563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d302e57946c42d799e9bf120873eba6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:52.789837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d302e57946c42d799e9bf120873eba6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:52.789837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24080c6-513b-4de7-b837-441d1d070a63 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d302e57946c42d799e9bf120873eba6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:52.806731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1207/2390] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:24:52 2022] DELETE /identity/v3/users/0d302e57946c42d799e9bf120873eba6 => generated 0 bytes in 317 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:53.135718 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:53.147747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0a4ada5-d59b-4dfe-8d4a-2bb7c5b24fde service placement] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:53.205001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:53.205862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.206154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.206472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] PATH_INFO: `/v3/projects/bb62d84584b3426687390833270648d6` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.208021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] RBAC: Authorizing `identity:delete_project(project_id=bb62d84584b3426687390833270648d6)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:53.210520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:53.339395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:53.351397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bb62d84584b3426687390833270648d6', 'request_id': 'req-15264c91-31f7-4bc1-b826-c28fc8d2b51b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:53.356841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bb62d84584b3426687390833270648d6', 'request_id': 'req-15264c91-31f7-4bc1-b826-c28fc8d2b51b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:53.437750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bb62d84584b3426687390833270648d6', 'request_id': 'req-15264c91-31f7-4bc1-b826-c28fc8d2b51b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:53.453863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] The token cache is being invalidate because project bb62d84584b3426687390833270648d6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:24:53.454153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:53.454844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-15264c91-31f7-4bc1-b826-c28fc8d2b51b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:24:53.456618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1184/2391] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:24:53 2022] DELETE /identity/v3/projects/bb62d84584b3426687390833270648d6 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:53.461740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e52a0b-f18f-4e3b-8dea-afc4fee69f39 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.462096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e52a0b-f18f-4e3b-8dea-afc4fee69f39 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.462437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e52a0b-f18f-4e3b-8dea-afc4fee69f39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.481481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-18e52a0b-f18f-4e3b-8dea-afc4fee69f39 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:53.482675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-18e52a0b-f18f-4e3b-8dea-afc4fee69f39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:53.544046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1208/2392] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:53.553105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fcd9d2-3aa4-41ca-847d-ca32b0c357e3 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.553424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fcd9d2-3aa4-41ca-847d-ca32b0c357e3 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.553718 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fcd9d2-3aa4-41ca-847d-ca32b0c357e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.573551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-60fcd9d2-3aa4-41ca-847d-ca32b0c357e3 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:53.574810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-60fcd9d2-3aa4-41ca-847d-ca32b0c357e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:53.583054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1185/2393] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:53.587418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e52a0b-f18f-4e3b-8dea-afc4fee69f39 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:53.597944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e52a0b-f18f-4e3b-8dea-afc4fee69f39 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:53.602663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:53.603355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.603640 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.603917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.604644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:24:53.604952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:53.606695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:53.619909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1209/2394] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:24:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:24:53.624208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fcd9d2-3aa4-41ca-847d-ca32b0c357e3 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:53.633585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fcd9d2-3aa4-41ca-847d-ca32b0c357e3 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:53.638231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:53.638916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.639204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.639551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.640179 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:53.642084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:53.752929 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1186/2395] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:24:53 2022] POST /identity/v3/projects => generated 379 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:53.757006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:53.768600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dda5a37-95fa-48a6-bb8d-e763a14a9641 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:53.828211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:53.828884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.829215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.829487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.830176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:53.832730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:53.929198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1210/2396] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:24:53 2022] POST /identity/v3/users => generated 290 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:53.933164 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:53.943873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07061ca-f5c3-4ed9-ac27-18d6b08ef115 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:53.948539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:53.949244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.949526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.949804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.950537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:53.952181 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:53.963262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1187/2397] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:24:53.967179 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:53.978208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1004556-9a34-42f6-a75c-6826d710a12a admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:53.983052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ceaf9a-950f-4845-8505-1b2c3aa418be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:53.983692 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ceaf9a-950f-4845-8505-1b2c3aa418be admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:53.983973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ceaf9a-950f-4845-8505-1b2c3aa418be admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:53.984249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ceaf9a-950f-4845-8505-1b2c3aa418be admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:53.984972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ceaf9a-950f-4845-8505-1b2c3aa418be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:53.986680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ceaf9a-950f-4845-8505-1b2c3aa418be admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.000318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1211/2398] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:53 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:54.004491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.014883 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08bdee9a-9f01-4166-92d2-81b1f7cc649a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.020189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:54.021066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.021351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.021634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] PATH_INFO: `/v3/projects/14e24315e4d34a0f89cccc3ac7c0a8f3/users/a6fec3ee45db4cf89a068961284e2850/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.037247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] RBAC: Authorizing `identity:create_grant(project_id=14e24315e4d34a0f89cccc3ac7c0a8f3, user_id=a6fec3ee45db4cf89a068961284e2850, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:54.040917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.134342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1188/2399] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:24:54 2022] PUT /identity/v3/projects/14e24315e4d34a0f89cccc3ac7c0a8f3/users/a6fec3ee45db4cf89a068961284e2850/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:54.146316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f56cf0-486e-436c-ae78-7e81573d496b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.146608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f56cf0-486e-436c-ae78-7e81573d496b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.146918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f56cf0-486e-436c-ae78-7e81573d496b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.171671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:54.174618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-14f56cf0-486e-436c-ae78-7e81573d496b None None] MFA Rules not processed for user `a6fec3ee45db4cf89a068961284e2850`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:54.176132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-14f56cf0-486e-436c-ae78-7e81573d496b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:54.241065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1212/2400] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:54 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:54.245498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.310156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de7ca48-9eaf-48ed-a8c0-ef2dc26c808a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.314745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:54.315421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.315715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.315974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.317186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:54.319115 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.387342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1189/2401] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:24:54 2022] POST /identity/v3/projects => generated 379 bytes in 142 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:54.391182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f56cf0-486e-436c-ae78-7e81573d496b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.402391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f56cf0-486e-436c-ae78-7e81573d496b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.456660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:54.457997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.458282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.458553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.459266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:54.461558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.543744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1213/2402] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:24:54 2022] POST /identity/v3/users => generated 296 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:24:54.545261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.562305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c04871-0fbb-4149-9871-ab14577e96f2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.567815 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:54.568479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.568769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.569053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.569793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:54.571509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.583629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1190/2403] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:54 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:54.588148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.598368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67ae76c-e752-46c5-8e70-3ce05e0a94bb admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.603359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:54.604274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.604551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.604848 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] PATH_INFO: `/v3/projects/d0447ade54c646f6bcdfd91c160585a7/users/32c9b525b9a94751bcd0e560cd059f33/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.620796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0447ade54c646f6bcdfd91c160585a7, user_id=32c9b525b9a94751bcd0e560cd059f33, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:54.623693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.683391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1214/2404] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:24:54 2022] PUT /identity/v3/projects/d0447ade54c646f6bcdfd91c160585a7/users/32c9b525b9a94751bcd0e560cd059f33/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 96 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:24:54.687585 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.698235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e028bcb3-d7ad-49a6-b360-51afb91bd664 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.730245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:54.759309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3c4a27-1665-4403-8490-19b9603cf93c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:54.759898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c4a27-1665-4403-8490-19b9603cf93c admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.760186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c4a27-1665-4403-8490-19b9603cf93c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.760467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3c4a27-1665-4403-8490-19b9603cf93c admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.761185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3c4a27-1665-4403-8490-19b9603cf93c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:54.762786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3c4a27-1665-4403-8490-19b9603cf93c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.773184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1191/2405] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:24:54 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:24:54.778309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.789557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4b8a5f-1f7d-4bcb-942e-7c13b5ef5891 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.795323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:54.808382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.808382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.808382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] PATH_INFO: `/v3/projects/d0447ade54c646f6bcdfd91c160585a7/users/32c9b525b9a94751bcd0e560cd059f33/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.808382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0447ade54c646f6bcdfd91c160585a7, user_id=32c9b525b9a94751bcd0e560cd059f33, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:54.808382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:54.815295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1215/2406] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:24:54 2022] PUT /identity/v3/projects/d0447ade54c646f6bcdfd91c160585a7/users/32c9b525b9a94751bcd0e560cd059f33/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:24:54.827170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710fbfdb-ec5e-4ae0-9159-4c569c77988f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:54.827534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710fbfdb-ec5e-4ae0-9159-4c569c77988f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:54.827914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710fbfdb-ec5e-4ae0-9159-4c569c77988f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:54.855042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-710fbfdb-ec5e-4ae0-9159-4c569c77988f None None] MFA Rules not processed for user `32c9b525b9a94751bcd0e560cd059f33`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:24:54.856257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-710fbfdb-ec5e-4ae0-9159-4c569c77988f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:54.925687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1192/2407] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:24:54 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:24:54.938153 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:54.940059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:24:54.996198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4193634-9e89-4724-b9cf-210ecc0bba21 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:54.999898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67015c0e-5a67-4b0b-b730-d3f7c9d83d08 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:55.000622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67015c0e-5a67-4b0b-b730-d3f7c9d83d08 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:55.000904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67015c0e-5a67-4b0b-b730-d3f7c9d83d08 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:55.001181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67015c0e-5a67-4b0b-b730-d3f7c9d83d08 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:55.015917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67015c0e-5a67-4b0b-b730-d3f7c9d83d08 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:55.015917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67015c0e-5a67-4b0b-b730-d3f7c9d83d08 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:55.018625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:24:55.025240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1216/2408] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:54 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:24:55.239836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710fbfdb-ec5e-4ae0-9159-4c569c77988f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:24:55.249450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-710fbfdb-ec5e-4ae0-9159-4c569c77988f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:24:55.252998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:24:55.253633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:24:55.253950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:24:55.254424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:24:55.312378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:24:55.313730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:24:55.322613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1193/2409] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:24:55 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:05.138799 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:05.318552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=77 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:05.408939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905cf23a-9805-475c-8353-b95290483b53 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:05.409228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905cf23a-9805-475c-8353-b95290483b53 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:05.409509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905cf23a-9805-475c-8353-b95290483b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:05.427598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-905cf23a-9805-475c-8353-b95290483b53 None None] MFA Rules not processed for user `d0adaa092d944202b3279d10e0f94a91`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:05.428796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-905cf23a-9805-475c-8353-b95290483b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:05.486139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1217/2410] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:05 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:05.808677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:05.810312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:05.869839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2728be3a-b1f7-4617-81ae-e96251cb8f35 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:05.873393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:05.874065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:05.881174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:05.881174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:05.881174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:05.885844 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:05.887346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:05.895468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1194/2411] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:05 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:06.255506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905cf23a-9805-475c-8353-b95290483b53 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:06.265278 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905cf23a-9805-475c-8353-b95290483b53 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:06.268809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54862c8-b1ee-4763-a49f-9bd789bbe205 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:06.269471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54862c8-b1ee-4763-a49f-9bd789bbe205 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:06.269779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54862c8-b1ee-4763-a49f-9bd789bbe205 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:06.270048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54862c8-b1ee-4763-a49f-9bd789bbe205 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:06.271831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c54862c8-b1ee-4763-a49f-9bd789bbe205 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:06.335590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54862c8-b1ee-4763-a49f-9bd789bbe205 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:06.337201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54862c8-b1ee-4763-a49f-9bd789bbe205 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:06.347386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1218/2412] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:06.858354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:06.871748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035e868b-72c1-430d-8c7b-0c618c05689a service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:06.876565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:06.877476 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:06.877785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:06.878065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] PATH_INFO: `/v3/users/a6fec3ee45db4cf89a068961284e2850` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:06.879347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6fec3ee45db4cf89a068961284e2850)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:06.881491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:07.067452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6fec3ee45db4cf89a068961284e2850'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:07.150400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6fec3ee45db4cf89a068961284e2850'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:07.151097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6fec3ee45db4cf89a068961284e2850'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:07.159218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6fec3ee45db4cf89a068961284e2850'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:07.160161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6fec3ee45db4cf89a068961284e2850'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:07.179004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1195/2413] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:25:06 2022] DELETE /identity/v3/users/a6fec3ee45db4cf89a068961284e2850 => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:07.184339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:07.184664 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:07.184957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:07.203089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:07.205183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:07.269511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1219/2414] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:07.280030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:07.281463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:07.340565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd1674c-ab0b-4314-8e14-1eea08f9a583 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:07.344237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:07.344887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:07.345169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:07.345449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:07.355515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:07.357040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:07.366832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1196/2415] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:07.872541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:07.874167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:07.891712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fb1f3ce-ec9f-4adb-bbca-f262b0883419 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:07.902422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:07.903347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:07.903507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:07.904348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] PATH_INFO: `/v3/projects/14e24315e4d34a0f89cccc3ac7c0a8f3` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:07.905483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14e24315e4d34a0f89cccc3ac7c0a8f3)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:07.907556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:08.052704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:08.072266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14e24315e4d34a0f89cccc3ac7c0a8f3', 'request_id': 'req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.173075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14e24315e4d34a0f89cccc3ac7c0a8f3', 'request_id': 'req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.180666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14e24315e4d34a0f89cccc3ac7c0a8f3', 'request_id': 'req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.191169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] The token cache is being invalidate because project 14e24315e4d34a0f89cccc3ac7c0a8f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:25:08.191461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.192125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.193687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1220/2416] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:25:07 2022] DELETE /identity/v3/projects/14e24315e4d34a0f89cccc3ac7c0a8f3 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:08.198250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:08.212094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91014f43-ef0a-4695-88bb-8ba5e989438e service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:08.221853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:08.276298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:08.277608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:08.278132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:08.278916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] PATH_INFO: `/v3/users/32c9b525b9a94751bcd0e560cd059f33` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:08.280222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=32c9b525b9a94751bcd0e560cd059f33)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:08.282373 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:08.454834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32c9b525b9a94751bcd0e560cd059f33'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.550368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32c9b525b9a94751bcd0e560cd059f33'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.551165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32c9b525b9a94751bcd0e560cd059f33'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.560614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32c9b525b9a94751bcd0e560cd059f33'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.561442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32c9b525b9a94751bcd0e560cd059f33'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:08.581120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1197/2417] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:25:08 2022] DELETE /identity/v3/users/32c9b525b9a94751bcd0e560cd059f33 => generated 0 bytes in 383 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:09.238472 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:09.243544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:09.255523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8ab3a4-e3a3-44a5-999b-fbd6069f2483 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:09.322965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:09.323995 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:09.324373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:09.324742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] PATH_INFO: `/v3/projects/d0447ade54c646f6bcdfd91c160585a7` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:09.325943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0447ade54c646f6bcdfd91c160585a7)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:09.329111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:09.442921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0447ade54c646f6bcdfd91c160585a7', 'request_id': 'req-33cf655e-0f6a-477c-b36e-ffecb639111f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:09.527537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0447ade54c646f6bcdfd91c160585a7', 'request_id': 'req-33cf655e-0f6a-477c-b36e-ffecb639111f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:09.530204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:09.531775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:09.538719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0447ade54c646f6bcdfd91c160585a7', 'request_id': 'req-33cf655e-0f6a-477c-b36e-ffecb639111f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:09.566755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] The token cache is being invalidate because project d0447ade54c646f6bcdfd91c160585a7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:25:09.566755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:09.566755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33cf655e-0f6a-477c-b36e-ffecb639111f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:09.578277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1221/2418] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:25:09 2022] DELETE /identity/v3/projects/d0447ade54c646f6bcdfd91c160585a7 => generated 0 bytes in 341 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:09.585349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12f8d2-a237-4a6c-a888-4681cc33fcb8 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:09.585796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12f8d2-a237-4a6c-a888-4681cc33fcb8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:09.586175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb12f8d2-a237-4a6c-a888-4681cc33fcb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:09.621085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bb12f8d2-a237-4a6c-a888-4681cc33fcb8 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:09.626569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bb12f8d2-a237-4a6c-a888-4681cc33fcb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:09.657327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3913560-a806-438b-bf00-a29a98f1d6ef admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:09.701766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=12,keystone:SELECT=86,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:09.702955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:09.723755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1222/2419] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:09.733732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8042bc68-7422-4bf2-a690-5c6105e3ddad None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:09.734064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8042bc68-7422-4bf2-a690-5c6105e3ddad None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:09.734644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8042bc68-7422-4bf2-a690-5c6105e3ddad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:09.757552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-8042bc68-7422-4bf2-a690-5c6105e3ddad None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:09.758913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-8042bc68-7422-4bf2-a690-5c6105e3ddad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:09.770035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1223/2420] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:09.774561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8042bc68-7422-4bf2-a690-5c6105e3ddad None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:09.779175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:09.779847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:09.780169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:09.780407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:09.784567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:09.804590 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8042bc68-7422-4bf2-a690-5c6105e3ddad None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:09.813227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:09.813587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:09.813873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:09.814711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:09.815117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:25:09.815534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:09.818498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:09.836128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1224/2421] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:25:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:25:09.840580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:09.850990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ff58aa-1b11-4b13-9be4-288b7929b211 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:09.857127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:09.857864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:09.858172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:09.858626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:09.859381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:09.861694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:09.877794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:09.883885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:09.898618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1198/2422] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:09 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:10.020183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1225/2423] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:25:09 2022] POST /identity/v3/projects => generated 365 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:25:10.020839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:10.064247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a84095-4bb5-46a6-be41-0f4801138a73 service nova] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:10.182572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:10.183308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:10.183588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:10.184034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:10.184737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:10.187079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:10.343306 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1199/2424] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:25:10 2022] POST /identity/v3/users => generated 283 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:25:10.343528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:10.348389 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5372ed-46f3-4836-80a1-8fff96f32781 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:10.355484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:10.356360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:10.356746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:10.357107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:10.357937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:10.359863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:10.372311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1226/2425] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:10 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:10.376122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:10.388707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39c0c66-cce2-4a82-ac98-4d2d629f6f43 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:10.399592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97abaf4b-244b-4906-b002-e24bbffca6f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:10.400652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97abaf4b-244b-4906-b002-e24bbffca6f5 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:10.401193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97abaf4b-244b-4906-b002-e24bbffca6f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:10.401674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97abaf4b-244b-4906-b002-e24bbffca6f5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:10.402621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97abaf4b-244b-4906-b002-e24bbffca6f5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:10.404332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97abaf4b-244b-4906-b002-e24bbffca6f5 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:10.419526 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1200/2426] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:10 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:10.420906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:10.433265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d81843c-18e4-4d76-900e-542f6c717468 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:10.441090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:10.445647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:10.445941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:10.446226 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] PATH_INFO: `/v3/projects/ad82b5b9f06546d394bf1274a7f4a747/users/5a7fc45067744c908df9fc61e80ad919/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:10.465575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad82b5b9f06546d394bf1274a7f4a747, user_id=5a7fc45067744c908df9fc61e80ad919, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:10.473296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:10.552945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1227/2427] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:25:10 2022] PUT /identity/v3/projects/ad82b5b9f06546d394bf1274a7f4a747/users/5a7fc45067744c908df9fc61e80ad919/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:10.564413 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:10.564950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:10.565882 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:10.598417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] MFA Rules not processed for user `5a7fc45067744c908df9fc61e80ad919`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:10.602350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:10.696832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:10.756525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1201/2428] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:10 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:10.761859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:10.868890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a68e6690-9709-4711-855b-b21aeba0ac63 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:10.876228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:10.876913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:10.877201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:10.877483 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:10.878181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:10.880858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:10.891735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:10.893199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:10.920300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e66305d-c26b-4dd1-9f5d-861c472b7e9e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:10.933211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:10.934296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:10.934683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:10.935097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] PATH_INFO: `/v3/users/d6da4b0616034aa1bb22c44a8f5ef280` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:10.936495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] RBAC: Authorizing `identity:delete_user(user_id=d6da4b0616034aa1bb22c44a8f5ef280)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:10.939404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.005190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1228/2429] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:25:10 2022] POST /identity/v3/projects => generated 363 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:25:11.009646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:11.088038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ba26f2-cf13-40e3-92e5-70412f6c6da0 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:11.092934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:11.093616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.093903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.094186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.095128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:11.096933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.106176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:11.176799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd6da4b0616034aa1bb22c44a8f5ef280'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:11.203550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1229/2430] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:25:11 2022] POST /identity/v3/users => generated 288 bytes in 194 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:25:11.207388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:11.225530 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5e6a8b-e8c4-488d-a019-b279938d043c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:11.231342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:11.232010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.232290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.232564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.233292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:11.234960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.247380 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd6da4b0616034aa1bb22c44a8f5ef280'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:11.248088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd6da4b0616034aa1bb22c44a8f5ef280'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:11.248220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1230/2431] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:11 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:11.252238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:11.254026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:11.256697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd6da4b0616034aa1bb22c44a8f5ef280'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:11.258476 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd6da4b0616034aa1bb22c44a8f5ef280'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:11.268357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5fc888-15ba-458e-9619-91ca99851119 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:11.273123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:11.275742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.276030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.276311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] PATH_INFO: `/v3/projects/4cb923f477fa40fab0fc58cd918686b0/users/5b61519777ac41a9b91c61a0d63b710d/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.285487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1202/2432] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:25:10 2022] DELETE /identity/v3/users/d6da4b0616034aa1bb22c44a8f5ef280 => generated 0 bytes in 394 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:25:11.297099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cb923f477fa40fab0fc58cd918686b0, user_id=5b61519777ac41a9b91c61a0d63b710d, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:11.299814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.367598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1231/2433] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:25:11 2022] PUT /identity/v3/projects/4cb923f477fa40fab0fc58cd918686b0/users/5b61519777ac41a9b91c61a0d63b710d/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:25:11.371996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:11.453125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-049391bb-0fb4-4d08-bae8-c7488789971a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:11.546129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0548ceaf-f424-4523-8fd2-3348596f4169 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:11.546129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0548ceaf-f424-4523-8fd2-3348596f4169 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.546129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0548ceaf-f424-4523-8fd2-3348596f4169 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.546444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0548ceaf-f424-4523-8fd2-3348596f4169 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.547450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0548ceaf-f424-4523-8fd2-3348596f4169 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:11.549265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0548ceaf-f424-4523-8fd2-3348596f4169 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.564283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:11.583007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1203/2434] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:11 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:11.587880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:11.601980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c13b8c-5fbd-4525-9f00-6b53664ee35a admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:11.609160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:11.610164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.610502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.610806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] PATH_INFO: `/v3/projects/4cb923f477fa40fab0fc58cd918686b0/users/5b61519777ac41a9b91c61a0d63b710d/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.615476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4cb923f477fa40fab0fc58cd918686b0, user_id=5b61519777ac41a9b91c61a0d63b710d, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:11.619184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.632300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1232/2435] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:25:11 2022] PUT /identity/v3/projects/4cb923f477fa40fab0fc58cd918686b0/users/5b61519777ac41a9b91c61a0d63b710d/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 45 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:11.643841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.644087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.644375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.676590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] MFA Rules not processed for user `5b61519777ac41a9b91c61a0d63b710d`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:11.677895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:11.692636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:11.694527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:11.772280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1204/2436] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:11 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:11.785347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:11.786749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:11.787379 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a34d7c9-8d0b-4afb-97c3-bd3a48a03524 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:11.798524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:11.799502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.799868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.800283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] PATH_INFO: `/v3/projects/33b09a1d039248aa95a089c8d8d99c68` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.801762 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] RBAC: Authorizing `identity:delete_project(project_id=33b09a1d039248aa95a089c8d8d99c68)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:11.803843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.857799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26f589bc-3ec3-4dd8-bdf6-baf1aec0ca9f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:11.861717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77b2b462-6232-45c2-8df8-649782c6e4ba service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:11.862231 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b2b462-6232-45c2-8df8-649782c6e4ba service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:11.862519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b2b462-6232-45c2-8df8-649782c6e4ba service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:11.862829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77b2b462-6232-45c2-8df8-649782c6e4ba service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:11.873029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b2b462-6232-45c2-8df8-649782c6e4ba service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:11.874569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77b2b462-6232-45c2-8df8-649782c6e4ba service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:11.889563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1205/2437] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:11 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:25:11.977479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33b09a1d039248aa95a089c8d8d99c68', 'request_id': 'req-9cb2cc14-5c1d-41b2-8184-058ef59a015f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:12.075836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33b09a1d039248aa95a089c8d8d99c68', 'request_id': 'req-9cb2cc14-5c1d-41b2-8184-058ef59a015f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:12.088533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33b09a1d039248aa95a089c8d8d99c68', 'request_id': 'req-9cb2cc14-5c1d-41b2-8184-058ef59a015f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:12.107451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] The token cache is being invalidate because project 33b09a1d039248aa95a089c8d8d99c68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:25:12.107678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:12.108498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:12.110813 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1233/2438] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:25:11 2022] DELETE /identity/v3/projects/33b09a1d039248aa95a089c8d8d99c68 => generated 0 bytes in 418 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:12.115540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ad9ef-04db-491e-8b41-44f33e89adc0 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.115834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ad9ef-04db-491e-8b41-44f33e89adc0 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.117520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705ad9ef-04db-491e-8b41-44f33e89adc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.135847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-705ad9ef-04db-491e-8b41-44f33e89adc0 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:12.138396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-705ad9ef-04db-491e-8b41-44f33e89adc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:12.177255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.178655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:12.215676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1206/2439] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:12.219451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:INSERT=5,keystone:SELECT=88,keystone:DELETE=7 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:12.224035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c30a83-0384-44c3-ad46-861bf24e0d6b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.224446 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c30a83-0384-44c3-ad46-861bf24e0d6b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.224831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3c30a83-0384-44c3-ad46-861bf24e0d6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.243809 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cb2cc14-5c1d-41b2-8184-058ef59a015f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.247515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.248170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.248614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.248781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.250578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:12.253839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-b3c30a83-0384-44c3-ad46-861bf24e0d6b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:12.255250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b3c30a83-0384-44c3-ad46-861bf24e0d6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:12.270935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1207/2440] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:25:12.273058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c30a83-0384-44c3-ad46-861bf24e0d6b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.282507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3c30a83-0384-44c3-ad46-861bf24e0d6b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.287698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.289120 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.289516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.289927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.290801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:25:12.291176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.292822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.308033 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1208/2441] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:25:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:25:12.312096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.320482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:12.323765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.323897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.324905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b99ebaa2-96d1-4610-9b92-11650976e2f2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.330829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.331496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.331781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.332058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.332790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.333211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1234/2442] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:12 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:12.334892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.415028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1209/2443] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:25:12 2022] POST /identity/v3/projects => generated 383 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:25:12.419468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.432480 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65eda205-dd1b-4a58-9643-3dc94d38a379 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.491701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.492390 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.492679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.493387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.493653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.495650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.603160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1235/2444] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:25:12 2022] POST /identity/v3/users => generated 292 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:25:12.607318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.619037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd112d9a-0304-4f3d-a90c-ec20fb5cfa1a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.625389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.630184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.630184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.630184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.630184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.630184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.640673 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1210/2445] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:25:12.650601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.661700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823541c3-c7e0-4fa0-806a-d24c72f00d23 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.667307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c1862b-8b01-4124-8ac6-ab7d5f1e73dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.667982 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c1862b-8b01-4124-8ac6-ab7d5f1e73dd admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.668625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c1862b-8b01-4124-8ac6-ab7d5f1e73dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.668853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c1862b-8b01-4124-8ac6-ab7d5f1e73dd admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.669614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c1862b-8b01-4124-8ac6-ab7d5f1e73dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.671249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c1862b-8b01-4124-8ac6-ab7d5f1e73dd admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.682251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1236/2446] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:12.686328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.696420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d79e16-0105-40ea-9c5b-9dcc3cdd2f10 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.701221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.702107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.702390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.702667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] PATH_INFO: `/v3/projects/cc4efb6abcf3412897b7f1484988aa2b/users/8d824401d1454a2bb19d7628e38b256a/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.735738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc4efb6abcf3412897b7f1484988aa2b, user_id=8d824401d1454a2bb19d7628e38b256a, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.738379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.803318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1211/2447] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:25:12 2022] PUT /identity/v3/projects/cc4efb6abcf3412897b7f1484988aa2b/users/8d824401d1454a2bb19d7628e38b256a/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:12.808542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c170ba4-edd9-4dfb-8e4d-ab27830c4d9e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.808892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c170ba4-edd9-4dfb-8e4d-ab27830c4d9e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.809176 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c170ba4-edd9-4dfb-8e4d-ab27830c4d9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.830105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-7c170ba4-edd9-4dfb-8e4d-ab27830c4d9e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:12.831463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7c170ba4-edd9-4dfb-8e4d-ab27830c4d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:12.891050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1237/2448] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:12.901283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:12.903515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:12.960166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f0d067-f8b8-4467-8415-fd1aa160f51f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:12.964747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:12.965414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:12.965733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:12.966014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:12.975692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:12.977260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:12.987098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1212/2449] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:18.825008 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9407c6-992d-4577-8537-bece09dcc53b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:18.825478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9407c6-992d-4577-8537-bece09dcc53b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:18.825607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a9407c6-992d-4577-8537-bece09dcc53b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:18.843181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:18.860001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0a9407c6-992d-4577-8537-bece09dcc53b None None] MFA Rules not processed for user `8d824401d1454a2bb19d7628e38b256a`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:18.861231 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0a9407c6-992d-4577-8537-bece09dcc53b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:18.929531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1238/2450] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:18 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:18.944397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:18.946913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:19.028304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c01201b-43d5-40a6-992e-be52135a649a service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:19.034959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:19.035885 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:19.036226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:19.036554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:19.046724 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:19.048431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:19.058727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1213/2451] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:21.462469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fac976-bde0-4278-99e5-23bd9abf4abf None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:21.463102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fac976-bde0-4278-99e5-23bd9abf4abf None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:21.463209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fac976-bde0-4278-99e5-23bd9abf4abf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:21.483258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c5fac976-bde0-4278-99e5-23bd9abf4abf None None] MFA Rules not processed for user `8d824401d1454a2bb19d7628e38b256a`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:21.489219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c5fac976-bde0-4278-99e5-23bd9abf4abf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:21.498864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1239/2452] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:25.825980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:25.825980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:25.848940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaaaf107-5d5b-40c4-969b-86b0d491e1a3 service nova] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:25.853177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:25.854043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:25.854583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:25.854948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] PATH_INFO: `/v3/users/5a7fc45067744c908df9fc61e80ad919` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:25.856810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a7fc45067744c908df9fc61e80ad919)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:25.858810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:26.036247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=93,keystone:INSERT=2,keystone:DELETE=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:26.051340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a7fc45067744c908df9fc61e80ad919'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:26.115304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a7fc45067744c908df9fc61e80ad919'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:26.115949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a7fc45067744c908df9fc61e80ad919'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:26.123756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a7fc45067744c908df9fc61e80ad919'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:26.130336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a7fc45067744c908df9fc61e80ad919'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:26.142196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1214/2453] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:25:25 2022] DELETE /identity/v3/users/5a7fc45067744c908df9fc61e80ad919 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:26.146591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:26.147119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:26.147240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:26.165384 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:26.168858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:26.226300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1240/2454] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:26.236683 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:26.238194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:26.299577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28246f6-75ad-491e-a55b-3b9e7b00a160 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:26.303471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:26.307655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:26.307655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:26.307655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:26.315729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:26.317343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:26.329324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1215/2455] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:26.872048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:26.873625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:26.886954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495779e-b285-44d5-8278-4120b69c9712 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:26.893595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:26.893595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:26.893595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:26.893595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] PATH_INFO: `/v3/projects/ad82b5b9f06546d394bf1274a7f4a747` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:26.894543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad82b5b9f06546d394bf1274a7f4a747)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:26.896761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:27.019276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad82b5b9f06546d394bf1274a7f4a747', 'request_id': 'req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.086725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad82b5b9f06546d394bf1274a7f4a747', 'request_id': 'req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.095367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad82b5b9f06546d394bf1274a7f4a747', 'request_id': 'req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.114315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] The token cache is being invalidate because project ad82b5b9f06546d394bf1274a7f4a747 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:25:27.115057 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.115957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.118587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1241/2456] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:25:26 2022] DELETE /identity/v3/projects/ad82b5b9f06546d394bf1274a7f4a747 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:27.122527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:27.143875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6d9eb6-540a-4510-a0d8-00267d858c4d service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:27.260499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:27.262416 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:27.263009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:27.263536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] PATH_INFO: `/v3/users/5b61519777ac41a9b91c61a0d63b710d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:27.266060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5b61519777ac41a9b91c61a0d63b710d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:27.269350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:27.427485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:27.431568 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:27.446931 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:27.449182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5b61519777ac41a9b91c61a0d63b710d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.453520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:27.499233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3e5e0e1-c4b1-4058-b3ef-a750a9ae77f9 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:27.503479 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:27.504637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:27.504940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:27.505283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:27.507782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:27.533527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5b61519777ac41a9b91c61a0d63b710d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.534149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5b61519777ac41a9b91c61a0d63b710d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.548373 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5b61519777ac41a9b91c61a0d63b710d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.550180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-50ffedf4-773c-4f0d-a024-abf0d494dc82 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5b61519777ac41a9b91c61a0d63b710d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:27.572547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1216/2457] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:25:27 2022] DELETE /identity/v3/users/5b61519777ac41a9b91c61a0d63b710d => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:27.584114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:27.586210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:27.596609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1242/2458] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:27 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:27.691752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:27.692337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:27.692690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:27.712051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:27.713355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:27.794846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1217/2459] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:25:27 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:27.806307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:27.807951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:27.964901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f252ea19-6506-4014-9631-319f320f0459 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:28.049954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7da396-0bd4-4a35-86ff-7a74bf900434 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:28.050330 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7da396-0bd4-4a35-86ff-7a74bf900434 service swift] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:28.050878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7da396-0bd4-4a35-86ff-7a74bf900434 service swift] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:28.051168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7da396-0bd4-4a35-86ff-7a74bf900434 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:28.062838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:28.064851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7da396-0bd4-4a35-86ff-7a74bf900434 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:28.067009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7da396-0bd4-4a35-86ff-7a74bf900434 service swift] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:28.077645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1243/2460] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:25:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:25:28.135768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:28.137427 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:28.313387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8923444-8f3e-4b76-a3cf-bf3d94f08b0e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:28.319546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:28.321457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:28.321988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:28.322491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] PATH_INFO: `/v3/projects/4cb923f477fa40fab0fc58cd918686b0` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:28.324095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4cb923f477fa40fab0fc58cd918686b0)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:28.326659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:28.466338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4cb923f477fa40fab0fc58cd918686b0', 'request_id': 'req-829d0050-be10-4ee2-b7fb-11f06ff4499c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:28.476343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4cb923f477fa40fab0fc58cd918686b0', 'request_id': 'req-829d0050-be10-4ee2-b7fb-11f06ff4499c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:28.591004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4cb923f477fa40fab0fc58cd918686b0', 'request_id': 'req-829d0050-be10-4ee2-b7fb-11f06ff4499c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:28.611334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] The token cache is being invalidate because project 4cb923f477fa40fab0fc58cd918686b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:25:28.611659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:28.612566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-829d0050-be10-4ee2-b7fb-11f06ff4499c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:25:28.614386 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1218/2461] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:25:28 2022] DELETE /identity/v3/projects/4cb923f477fa40fab0fc58cd918686b0 => generated 0 bytes in 479 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:28.620373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53930189-1ddf-4a9a-9927-3436df89f075 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:28.620915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53930189-1ddf-4a9a-9927-3436df89f075 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:28.620915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53930189-1ddf-4a9a-9927-3436df89f075 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:28.647178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-53930189-1ddf-4a9a-9927-3436df89f075 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:28.649243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-53930189-1ddf-4a9a-9927-3436df89f075 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:28.746229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1244/2462] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:28.754176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896b7d0c-0057-4302-95b2-fe3075e5d4d7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:28.754467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896b7d0c-0057-4302-95b2-fe3075e5d4d7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:28.754750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896b7d0c-0057-4302-95b2-fe3075e5d4d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:28.773806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:28.786533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-896b7d0c-0057-4302-95b2-fe3075e5d4d7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:28.787837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-896b7d0c-0057-4302-95b2-fe3075e5d4d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:28.799802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1219/2463] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:28.801132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53930189-1ddf-4a9a-9927-3436df89f075 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:28.811649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53930189-1ddf-4a9a-9927-3436df89f075 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:28.816508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:28.817182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:28.819728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:28.820069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:28.821333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:25:28.821684 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:28.823430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:28.846815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1245/2464] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:25:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 46 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:25:28.851297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896b7d0c-0057-4302-95b2-fe3075e5d4d7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:28.864707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896b7d0c-0057-4302-95b2-fe3075e5d4d7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:28.870981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:28.871842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:28.872232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:28.872627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:28.873834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:28.876102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:28.992953 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1220/2465] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:25:28 2022] POST /identity/v3/projects => generated 377 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:25:28.997111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:29.015821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b03ebdd-3eaf-4627-9c3f-fff419623455 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:29.101051 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:29.102085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:29.103043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:29.103425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:29.104234 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:29.106806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:29.241544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1246/2466] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:25:28 2022] POST /identity/v3/users => generated 289 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:25:29.242713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:29.254288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e24fcfc-f8ee-4bc6-b52b-2e2e9367539a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:29.260159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:29.260935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:29.261298 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:29.261649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:29.262653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:29.264447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:29.280826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1221/2467] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:29 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:29.284727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:29.300761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880f3a32-1848-419c-839c-98481fb6fedc admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:29.314361 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc160a0-1903-42e0-9e69-68a6230752ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:29.315082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc160a0-1903-42e0-9e69-68a6230752ae admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:29.315368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc160a0-1903-42e0-9e69-68a6230752ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:29.315641 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc160a0-1903-42e0-9e69-68a6230752ae admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:29.316373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc160a0-1903-42e0-9e69-68a6230752ae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:29.318286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc160a0-1903-42e0-9e69-68a6230752ae admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:29.343826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1247/2468] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:29 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:29.345326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:29.366642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf481cfc-a352-4562-a41b-2d53f0929af4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:29.385968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:29.387041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:29.387193 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:29.387458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] PATH_INFO: `/v3/projects/eb2ede8d155548089f180286ca870a3c/users/2c21caa92f31480492abd97e0e30d592/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:29.410477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb2ede8d155548089f180286ca870a3c, user_id=2c21caa92f31480492abd97e0e30d592, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:29.414029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:29.484614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1222/2469] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:25:29 2022] PUT /identity/v3/projects/eb2ede8d155548089f180286ca870a3c/users/2c21caa92f31480492abd97e0e30d592/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:29.496323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea1cb5-179d-408d-8d5e-60cb9d0cd8f5 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:29.496678 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea1cb5-179d-408d-8d5e-60cb9d0cd8f5 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:29.496906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ea1cb5-179d-408d-8d5e-60cb9d0cd8f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:29.530464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f1ea1cb5-179d-408d-8d5e-60cb9d0cd8f5 None None] MFA Rules not processed for user `2c21caa92f31480492abd97e0e30d592`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:29.531579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f1ea1cb5-179d-408d-8d5e-60cb9d0cd8f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:29.610012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:29.637286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1248/2470] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:29 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:29.639206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:29.756304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3354468-06bc-4cfb-9663-add4d60e5418 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:29.761971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:29.763458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:29.763458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:29.763458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:29.764501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:29.766692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:29.858811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1223/2471] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:25:29 2022] POST /identity/v3/projects => generated 377 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:25:29.862017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ea1cb5-179d-408d-8d5e-60cb9d0cd8f5 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:29.876441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ea1cb5-179d-408d-8d5e-60cb9d0cd8f5 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:30.012759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:30.013631 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:30.014063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:30.014741 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:30.015630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:30.019504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:30.140168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1249/2472] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:25:29 2022] POST /identity/v3/users => generated 295 bytes in 280 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:25:30.143985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:30.160168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b5946af-b0ab-41b8-b5c0-ecd16aa9e898 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:30.171868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:30.172567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:30.173754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:30.174047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:30.176695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:30.179065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:30.194274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1224/2473] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:30 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:30.198970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:30.225667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f781287-68bb-4e8b-b60c-a2d888385619 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:30.232434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:30.233456 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:30.236316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:30.236316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] PATH_INFO: `/v3/projects/aecf5eb629674d4b975526bd332a745e/users/71edd006f3b84aa78ecf5014744617f8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:30.254243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=aecf5eb629674d4b975526bd332a745e, user_id=71edd006f3b84aa78ecf5014744617f8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:30.256958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:30.364140 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1250/2474] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:25:30 2022] PUT /identity/v3/projects/aecf5eb629674d4b975526bd332a745e/users/71edd006f3b84aa78ecf5014744617f8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:30.368580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:30.383019 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b51302-c4d8-4099-95d9-e7c8b74e0080 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:30.466497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70aa7c2b-6f4b-4c67-9d16-96ffa41356db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:30.467352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aa7c2b-6f4b-4c67-9d16-96ffa41356db admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:30.467762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aa7c2b-6f4b-4c67-9d16-96ffa41356db admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:30.468140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70aa7c2b-6f4b-4c67-9d16-96ffa41356db admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:30.468978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70aa7c2b-6f4b-4c67-9d16-96ffa41356db admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:30.471141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70aa7c2b-6f4b-4c67-9d16-96ffa41356db admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:30.489045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1225/2475] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:25:30 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:25:30.494586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:30.506501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c1aeb0-a139-4e91-a50b-f54cb15316bb admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:30.513999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:30.514950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:30.515255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:30.515682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] PATH_INFO: `/v3/projects/aecf5eb629674d4b975526bd332a745e/users/71edd006f3b84aa78ecf5014744617f8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:30.518182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aecf5eb629674d4b975526bd332a745e, user_id=71edd006f3b84aa78ecf5014744617f8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:30.524046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:30.536943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1251/2476] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:25:30 2022] PUT /identity/v3/projects/aecf5eb629674d4b975526bd332a745e/users/71edd006f3b84aa78ecf5014744617f8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:25:30.556529 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:30.558221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:30.558726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:30.585653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:30.596911 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] MFA Rules not processed for user `71edd006f3b84aa78ecf5014744617f8`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:30.599116 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:30.713702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1226/2477] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:25:30 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:25:30.732600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:30.734983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:30.842441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5174d2c9-b4bb-4299-bde8-079d34513b01 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:30.847545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:30.848547 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:30.848832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:30.849114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:30.926087 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:30.930635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:30.938588 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:30.946052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1252/2478] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:30 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:37.486102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:37.488107 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:37.505736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453a4861-0dec-4c15-8f40-8fb2c310828e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:37.509562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea99e3e-1ac2-4cea-82bb-6df0881b0c43 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:37.510424 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea99e3e-1ac2-4cea-82bb-6df0881b0c43 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:37.510806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea99e3e-1ac2-4cea-82bb-6df0881b0c43 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:37.511150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea99e3e-1ac2-4cea-82bb-6df0881b0c43 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:37.521253 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea99e3e-1ac2-4cea-82bb-6df0881b0c43 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:37.523613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea99e3e-1ac2-4cea-82bb-6df0881b0c43 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:37.535612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1227/2479] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:37 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:40.983569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:44.427517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:44.429022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:44.488237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f275478-2998-45f8-a9fa-bc763e844451 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:44.491983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeae7793-afe2-49fe-9eb2-0b6d5c44f805 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:44.493271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeae7793-afe2-49fe-9eb2-0b6d5c44f805 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:44.493556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeae7793-afe2-49fe-9eb2-0b6d5c44f805 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:44.493966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeae7793-afe2-49fe-9eb2-0b6d5c44f805 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:44.495880 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-aeae7793-afe2-49fe-9eb2-0b6d5c44f805 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:44.558993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeae7793-afe2-49fe-9eb2-0b6d5c44f805 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:44.560460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeae7793-afe2-49fe-9eb2-0b6d5c44f805 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:44.569806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1253/2480] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:44 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:47.530235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=35 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:54.565393 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=48 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:25:55.044059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69686fb8-70c2-4ec7-b62d-d3ad20c49029 None None] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:55.044059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69686fb8-70c2-4ec7-b62d-d3ad20c49029 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:55.044059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69686fb8-70c2-4ec7-b62d-d3ad20c49029 None None] PATH_INFO: `/` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:55.044059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1228/2481] 10.222.0.27 () {54 vars in 921 bytes} [Thu Jul 14 12:25:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 14 12:25:55.048864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c747e8-97a7-4504-9ab9-d1011e765d04 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:55.049149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c747e8-97a7-4504-9ab9-d1011e765d04 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:55.049431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c747e8-97a7-4504-9ab9-d1011e765d04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:55.067260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d3c747e8-97a7-4504-9ab9-d1011e765d04 None None] MFA Rules not processed for user `da24a7ecdcc249f0820add0404a7856c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:25:55.068472 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d3c747e8-97a7-4504-9ab9-d1011e765d04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:55.137202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1254/2482] 10.222.0.27 () {58 vars in 1018 bytes} [Thu Jul 14 12:25:55 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:25:55.149606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69686fb8-70c2-4ec7-b62d-d3ad20c49029 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:55.151363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-69686fb8-70c2-4ec7-b62d-d3ad20c49029 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:55.222852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69686fb8-70c2-4ec7-b62d-d3ad20c49029 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:55.226552 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd2b0127-e84a-4824-8c5e-fd81aa838187 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:55.227256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b0127-e84a-4824-8c5e-fd81aa838187 service glance] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:55.227575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b0127-e84a-4824-8c5e-fd81aa838187 service glance] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:55.227965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd2b0127-e84a-4824-8c5e-fd81aa838187 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:55.238689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2b0127-e84a-4824-8c5e-fd81aa838187 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:55.240180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd2b0127-e84a-4824-8c5e-fd81aa838187 service glance] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:55.252270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1229/2483] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:55 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:25:57.139598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c747e8-97a7-4504-9ab9-d1011e765d04 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:25:57.148817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c747e8-97a7-4504-9ab9-d1011e765d04 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:25:57.152468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:25:57.153127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:25:57.153401 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:25:57.153676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:25:57.155604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:25:57.224798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:25:57.226257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:25:57.237587 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1255/2484] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:25:57 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:00.913249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:00.913713 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:00.913822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:00.935146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:00.936269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:01.010074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1230/2485] 10.222.0.27 () {58 vars in 1009 bytes} [Thu Jul 14 12:26:00 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:01.034908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:01.036932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:01.157155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22eb34da-f3bb-4938-b530-a6fc5e15e78e service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:01.162539 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:01.163235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:01.163521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:01.163800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:01.176859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:01.179269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:01.189365 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1256/2486] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:26:01 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:26:05.784333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:05.785938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:05.806587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c18b95c-986c-44b8-99f9-c1d91b9ab574 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:05.811356 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:05.812545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:05.816068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:05.816068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] PATH_INFO: `/v3/users/7489ed79115044388225314714876f24` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:05.816068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7489ed79115044388225314714876f24)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:05.817199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:06.015385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:06.037290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19a184a1-e825-4098-aa5f-4ca1efff3e38 service swift] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:06.044163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:06.051986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7489ed79115044388225314714876f24'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.055895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:06.057121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:06.057592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] PATH_INFO: `/v3/users/2c21caa92f31480492abd97e0e30d592` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:06.061135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2c21caa92f31480492abd97e0e30d592)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:06.074043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:06.115130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:06.118437 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7489ed79115044388225314714876f24'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.119372 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7489ed79115044388225314714876f24'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.129300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7489ed79115044388225314714876f24'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.143426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7489ed79115044388225314714876f24'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.178699 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1231/2487] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:05 2022] DELETE /identity/v3/users/7489ed79115044388225314714876f24 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:06.375092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2c21caa92f31480492abd97e0e30d592'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.458555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:06.460091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:06.474346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2c21caa92f31480492abd97e0e30d592'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.488158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2c21caa92f31480492abd97e0e30d592'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.490758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2c21caa92f31480492abd97e0e30d592'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.493394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30475379-5274-42e0-8330-02bda88caa92 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2c21caa92f31480492abd97e0e30d592'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.495664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:06.523441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1257/2488] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:06 2022] DELETE /identity/v3/users/2c21caa92f31480492abd97e0e30d592 => generated 0 bytes in 505 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:06.525436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:06.525733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:06.526227 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:06.545944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:06.547232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:06.552813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-061e6954-098e-4932-b6aa-e368dfe4e3e1 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:06.630150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1258/2489] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:06.645438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:06.650637 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:06.652631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:06.653895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:06.654215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:06.654574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] PATH_INFO: `/v3/projects/dfa18dfa445442d4be02d187ed24325f` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:06.655904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dfa18dfa445442d4be02d187ed24325f)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:06.657968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:06.735771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d663c824-c89e-4f35-81ef-79a33781a62e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:06.739386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:06.740036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:06.740317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:06.740594 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:06.752956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:06.754889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:06.763968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dfa18dfa445442d4be02d187ed24325f', 'request_id': 'req-707befaa-7451-4243-8500-e2ce61a07bb5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.766583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1259/2490] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:06.770859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dfa18dfa445442d4be02d187ed24325f', 'request_id': 'req-707befaa-7451-4243-8500-e2ce61a07bb5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.839819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dfa18dfa445442d4be02d187ed24325f', 'request_id': 'req-707befaa-7451-4243-8500-e2ce61a07bb5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.852762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] The token cache is being invalidate because project dfa18dfa445442d4be02d187ed24325f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:06.853195 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.853981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:06.855948 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1232/2491] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:06 2022] DELETE /identity/v3/projects/dfa18dfa445442d4be02d187ed24325f => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:26:06.860034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:06.861521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:06.875935 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bec373f-b827-4142-a5df-1a37b68adc6e service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:06.908718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:06.938735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:06.939635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:06.939966 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:06.940246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] PATH_INFO: `/v3/users/1aec8c73c2dd4021b4740fbad1f51adc` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:06.941511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=1aec8c73c2dd4021b4740fbad1f51adc)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:06.943539 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:07.141711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1aec8c73c2dd4021b4740fbad1f51adc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.237929 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1aec8c73c2dd4021b4740fbad1f51adc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.247595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1aec8c73c2dd4021b4740fbad1f51adc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.248373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1aec8c73c2dd4021b4740fbad1f51adc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.249006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1aec8c73c2dd4021b4740fbad1f51adc'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.273746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1260/2492] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:06 2022] DELETE /identity/v3/users/1aec8c73c2dd4021b4740fbad1f51adc => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:07.348950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:07.350399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:07.364822 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707befaa-7451-4243-8500-e2ce61a07bb5 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:07.433157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:07.442267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:07.443292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:07.451114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:07.451468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] PATH_INFO: `/v3/projects/eb2ede8d155548089f180286ca870a3c` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:07.455042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb2ede8d155548089f180286ca870a3c)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:07.457111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:07.559866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb2ede8d155548089f180286ca870a3c', 'request_id': 'req-14e5f62e-cf44-4209-a03a-a251936d57f1'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.565143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb2ede8d155548089f180286ca870a3c', 'request_id': 'req-14e5f62e-cf44-4209-a03a-a251936d57f1'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.577732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:07.579198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:07.592882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63bad444-9270-49a6-bb13-e0d93957fc7b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:07.682409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:07.683438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:07.683722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:07.683999 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] PATH_INFO: `/v3/projects/8d43e94a04ab44dda0f87babe90a02be` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:07.685200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d43e94a04ab44dda0f87babe90a02be)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:07.687264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:07.695057 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb2ede8d155548089f180286ca870a3c', 'request_id': 'req-14e5f62e-cf44-4209-a03a-a251936d57f1'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.716919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] The token cache is being invalidate because project eb2ede8d155548089f180286ca870a3c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:07.717238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.717902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.720897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1233/2493] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:07 2022] DELETE /identity/v3/projects/eb2ede8d155548089f180286ca870a3c => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:07.724523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:07.730914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:07.740342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e5f62e-cf44-4209-a03a-a251936d57f1 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:07.745920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:07.747900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:07.748251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:07.748696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] PATH_INFO: `/v3/users/71edd006f3b84aa78ecf5014744617f8` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:07.750026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] RBAC: Authorizing `identity:delete_user(user_id=71edd006f3b84aa78ecf5014744617f8)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:07.752088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:07.887802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d43e94a04ab44dda0f87babe90a02be', 'request_id': 'req-df78fbb7-288c-4820-acc0-72a61a0d2c49'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.892564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:07.971781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d43e94a04ab44dda0f87babe90a02be', 'request_id': 'req-df78fbb7-288c-4820-acc0-72a61a0d2c49'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.980942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d43e94a04ab44dda0f87babe90a02be', 'request_id': 'req-df78fbb7-288c-4820-acc0-72a61a0d2c49'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.995994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] The token cache is being invalidate because project 8d43e94a04ab44dda0f87babe90a02be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:07.996290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.996994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:07.998620 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1261/2494] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:07 2022] DELETE /identity/v3/projects/8d43e94a04ab44dda0f87babe90a02be => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:08.074133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71edd006f3b84aa78ecf5014744617f8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:08.142802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71edd006f3b84aa78ecf5014744617f8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:08.143572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71edd006f3b84aa78ecf5014744617f8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:08.163914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71edd006f3b84aa78ecf5014744617f8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:08.166531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9c4b1275-d264-4007-a435-47c19486770d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71edd006f3b84aa78ecf5014744617f8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:08.197548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1234/2495] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:07 2022] DELETE /identity/v3/users/71edd006f3b84aa78ecf5014744617f8 => generated 0 bytes in 470 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:26:08.728199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:08.729979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:08.808614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df78fbb7-288c-4820-acc0-72a61a0d2c49 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:08.875586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:08.876588 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:08.876872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:08.877150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] PATH_INFO: `/v3/projects/aecf5eb629674d4b975526bd332a745e` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:08.878426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aecf5eb629674d4b975526bd332a745e)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:08.880600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:09.001313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aecf5eb629674d4b975526bd332a745e', 'request_id': 'req-30be714c-147e-4202-963c-e2b1e71c5836'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:09.066859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aecf5eb629674d4b975526bd332a745e', 'request_id': 'req-30be714c-147e-4202-963c-e2b1e71c5836'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:09.075286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aecf5eb629674d4b975526bd332a745e', 'request_id': 'req-30be714c-147e-4202-963c-e2b1e71c5836'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:09.087503 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] The token cache is being invalidate because project aecf5eb629674d4b975526bd332a745e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:09.087844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:09.088517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-30be714c-147e-4202-963c-e2b1e71c5836 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:09.090052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1262/2496] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:08 2022] DELETE /identity/v3/projects/aecf5eb629674d4b975526bd332a745e => generated 0 bytes in 363 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:09.094984 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a05b8-aee0-465f-9118-979f6dc2b2f7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.095444 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a05b8-aee0-465f-9118-979f6dc2b2f7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.095559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933a05b8-aee0-465f-9118-979f6dc2b2f7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.113797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-933a05b8-aee0-465f-9118-979f6dc2b2f7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:09.115091 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-933a05b8-aee0-465f-9118-979f6dc2b2f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:09.170279 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:09.179098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1235/2497] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:26:09.187215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f3e102-9253-4c4d-84e8-eb72f1ac4a31 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.187492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f3e102-9253-4c4d-84e8-eb72f1ac4a31 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.187774 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f3e102-9253-4c4d-84e8-eb72f1ac4a31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.208098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d5f3e102-9253-4c4d-84e8-eb72f1ac4a31 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:09.209490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d5f3e102-9253-4c4d-84e8-eb72f1ac4a31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:09.218490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1263/2498] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:09.222569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a05b8-aee0-465f-9118-979f6dc2b2f7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:09.236240 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933a05b8-aee0-465f-9118-979f6dc2b2f7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:09.287776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:09.287776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.287776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.287776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.289014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:26:09.289014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:09.289014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:09.289014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1236/2499] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:26:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:26:09.294857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f3e102-9253-4c4d-84e8-eb72f1ac4a31 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:09.304517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f3e102-9253-4c4d-84e8-eb72f1ac4a31 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:09.336910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:09.337627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.337910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.338217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.338950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:09.340883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:09.354596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:09.608039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1264/2500] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:09 2022] POST /identity/v3/projects => generated 367 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:09.612538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:09.624166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560c45af-54d3-4629-81b2-4386dde86d44 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:09.690938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:09.691854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.692143 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.692422 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.693191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:09.695173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:09.800180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1237/2501] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:09 2022] POST /identity/v3/users => generated 284 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:09.804667 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:09.816518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f2d452-4e10-46ba-b360-1b1d521d8ffe admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:09.821810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:09.823356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.823356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.823356 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.824017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:09.825635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:09.836585 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1265/2502] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:09.841398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:09.854679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72a04a09-6650-465c-a850-31c134b554d9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:09.859711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e99c3c9d-7d40-49da-aa5d-7dd2b5061cc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:09.860391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c3c9d-7d40-49da-aa5d-7dd2b5061cc6 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.860677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c3c9d-7d40-49da-aa5d-7dd2b5061cc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.860956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e99c3c9d-7d40-49da-aa5d-7dd2b5061cc6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.861697 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c3c9d-7d40-49da-aa5d-7dd2b5061cc6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:09.863363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e99c3c9d-7d40-49da-aa5d-7dd2b5061cc6 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:09.875017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1238/2503] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:09 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:09.879043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:09.890444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fb3aa51-4641-4d11-a7c4-53d59247918d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:09.895283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:09.896175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:09.896451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:09.896723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] PATH_INFO: `/v3/projects/27614050356a4b23988dadb369f2e31a/users/e365f83c7b074f3fbc3e90c765db8c09/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:09.913214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] RBAC: Authorizing `identity:create_grant(project_id=27614050356a4b23988dadb369f2e31a, user_id=e365f83c7b074f3fbc3e90c765db8c09, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:09.916468 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:09.987219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1266/2504] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:09 2022] PUT /identity/v3/projects/27614050356a4b23988dadb369f2e31a/users/e365f83c7b074f3fbc3e90c765db8c09/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:09.999799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ad216-96e6-4dfe-9b19-e2bba4e938fe None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.000124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ad216-96e6-4dfe-9b19-e2bba4e938fe None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.000440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1ad216-96e6-4dfe-9b19-e2bba4e938fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.029043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ed1ad216-96e6-4dfe-9b19-e2bba4e938fe None None] MFA Rules not processed for user `e365f83c7b074f3fbc3e90c765db8c09`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:10.034053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ed1ad216-96e6-4dfe-9b19-e2bba4e938fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:10.111406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1239/2505] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:09 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:10.115860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:10.183964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0065eed4-cdc0-461d-8202-9111107e489b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:10.192565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:10.193252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.193538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.193815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.194523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:10.196495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:10.469786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1267/2506] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:10 2022] POST /identity/v3/projects => generated 369 bytes in 354 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:10.473853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ad216-96e6-4dfe-9b19-e2bba4e938fe None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:10.485194 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1ad216-96e6-4dfe-9b19-e2bba4e938fe None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:10.537979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:10.545874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:10.546545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.546861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.547141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.547834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:10.549770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:10.636017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1240/2507] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:10 2022] POST /identity/v3/users => generated 291 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:10.639138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:10.649293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3410e51f-609f-4e3c-8279-ea8f3cc524be admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:10.654342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:10.655036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.655315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.655592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.656308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:10.657920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:10.669274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1268/2508] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:10.675174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:10.686330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8a2ed83-306f-4e48-88bb-3e6a17f904fa admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:10.691686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:10.692572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.692850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.693124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] PATH_INFO: `/v3/projects/58ed78ecfebf491090728db5dfe691b4/users/51edc434ecbf49b59ff6ea0b493929c0/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.709318 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58ed78ecfebf491090728db5dfe691b4, user_id=51edc434ecbf49b59ff6ea0b493929c0, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:10.712073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:10.777710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1241/2509] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:10 2022] PUT /identity/v3/projects/58ed78ecfebf491090728db5dfe691b4/users/51edc434ecbf49b59ff6ea0b493929c0/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:10.783156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:10.793853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd383de8-69bf-4447-b254-0f848e129108 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:10.851905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a002a6-53ef-4524-bdaf-87632343f95b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:10.852622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a002a6-53ef-4524-bdaf-87632343f95b admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.852844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a002a6-53ef-4524-bdaf-87632343f95b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.853121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a002a6-53ef-4524-bdaf-87632343f95b admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.854509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a002a6-53ef-4524-bdaf-87632343f95b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:10.856196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a002a6-53ef-4524-bdaf-87632343f95b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:10.866740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1269/2510] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:10 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:26:10.870218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:10.880858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530b9e3f-820e-4168-be61-37fc7b344ab3 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:10.886060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:10.886976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.887258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.887532 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] PATH_INFO: `/v3/projects/58ed78ecfebf491090728db5dfe691b4/users/51edc434ecbf49b59ff6ea0b493929c0/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.889555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] RBAC: Authorizing `identity:create_grant(project_id=58ed78ecfebf491090728db5dfe691b4, user_id=51edc434ecbf49b59ff6ea0b493929c0, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:10.892256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:10.903694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1242/2511] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:10 2022] PUT /identity/v3/projects/58ed78ecfebf491090728db5dfe691b4/users/51edc434ecbf49b59ff6ea0b493929c0/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:10.913943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:10.914237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:10.914518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:10.941564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] MFA Rules not processed for user `51edc434ecbf49b59ff6ea0b493929c0`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:10.942884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:10.954939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:11.010165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1270/2512] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:10 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:26:11.021419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:11.023144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:11.080219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1895d532-3b7f-45da-9507-ae13a1a8e046 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:11.083898 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77f8c456-5b14-42f9-8544-9c738123209b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:11.084540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8c456-5b14-42f9-8544-9c738123209b service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:11.084821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8c456-5b14-42f9-8544-9c738123209b service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:11.085098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77f8c456-5b14-42f9-8544-9c738123209b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:11.094646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8c456-5b14-42f9-8544-9c738123209b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:11.099497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77f8c456-5b14-42f9-8544-9c738123209b service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:11.105527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1243/2513] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:11 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:11.231628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:11.233020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:11.245164 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98acad51-887a-4e11-a179-1f28f1e50c66 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:11.248777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:11.249407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:11.249688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:11.249960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:11.305316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:11.307327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:11.315983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1271/2514] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:11 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:16.628519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c788bc-1841-43e2-a458-33fcc8ddd963 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:16.629056 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c788bc-1841-43e2-a458-33fcc8ddd963 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:16.629157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c788bc-1841-43e2-a458-33fcc8ddd963 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:16.672175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-d9c788bc-1841-43e2-a458-33fcc8ddd963 None None] MFA Rules not processed for user `1843ddb84e8b434dbed5ebd32419b8d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:16.672175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d9c788bc-1841-43e2-a458-33fcc8ddd963 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:16.777170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1244/2515] 10.222.0.27 () {58 vars in 967 bytes} [Thu Jul 14 12:26:16 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:16.793175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:16.793988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:16.812191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c788bc-1841-43e2-a458-33fcc8ddd963 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:16.828068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:16.908025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6809670-99ca-4fa8-bf33-30dc168e24aa service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:16.911351 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c788bc-1841-43e2-a458-33fcc8ddd963 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:16.914415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aaf1efe-f2fb-430a-8d51-4aea42131bf2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '7b1634a6e6ec4db9a8851b1329ffe6a0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:16.914415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf1efe-f2fb-430a-8d51-4aea42131bf2 service swift] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:16.914657 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf1efe-f2fb-430a-8d51-4aea42131bf2 service swift] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:16.914860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aaf1efe-f2fb-430a-8d51-4aea42131bf2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:16.916959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:16.918129 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:16.918412 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:16.918688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] PATH_INFO: `/v3/users/e365f83c7b074f3fbc3e90c765db8c09` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:16.920008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e365f83c7b074f3fbc3e90c765db8c09)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:16.921944 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:16.924283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaf1efe-f2fb-430a-8d51-4aea42131bf2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:16.925876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aaf1efe-f2fb-430a-8d51-4aea42131bf2 service swift] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:16.934596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1272/2516] 10.222.0.27 () {60 vars in 1384 bytes} [Thu Jul 14 12:26:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 14 12:26:17.098559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e365f83c7b074f3fbc3e90c765db8c09'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:17.176707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e365f83c7b074f3fbc3e90c765db8c09'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:17.177387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e365f83c7b074f3fbc3e90c765db8c09'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:17.185614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e365f83c7b074f3fbc3e90c765db8c09'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:17.186278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e365f83c7b074f3fbc3e90c765db8c09'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:17.203468 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1245/2517] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:16 2022] DELETE /identity/v3/users/e365f83c7b074f3fbc3e90c765db8c09 => generated 0 bytes in 392 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:17.207877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:17.208172 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:17.208448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:17.219034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:17.247056 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:17.248232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:17.307237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1273/2518] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:17.316795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:17.318313 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:17.376854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa663a-67ed-44c4-acc7-440b5b40fb57 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:17.380654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:17.381294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:17.381574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:17.381847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:17.391071 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:17.392489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:17.402463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1246/2519] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:17.852498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:17.855073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:17.875347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a84ac16f-5750-47ee-bf04-22f0995696dd None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:17.878216 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:17.879174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:17.879454 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:17.879727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] PATH_INFO: `/v3/projects/27614050356a4b23988dadb369f2e31a` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:17.881560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=27614050356a4b23988dadb369f2e31a)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:17.884042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:18.038229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '27614050356a4b23988dadb369f2e31a', 'request_id': 'req-035bc10f-a31c-4783-abd9-17ebb9568fa6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.107284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '27614050356a4b23988dadb369f2e31a', 'request_id': 'req-035bc10f-a31c-4783-abd9-17ebb9568fa6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.114431 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '27614050356a4b23988dadb369f2e31a', 'request_id': 'req-035bc10f-a31c-4783-abd9-17ebb9568fa6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.126107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] The token cache is being invalidate because project 27614050356a4b23988dadb369f2e31a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:18.126395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.127064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.131873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1274/2520] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:17 2022] DELETE /identity/v3/projects/27614050356a4b23988dadb369f2e31a => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:18.132946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:18.143341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5b1b99-bd7f-4691-87f9-df57aa814ce0 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:18.147970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:18.202758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:18.203684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:18.203945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:18.204212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] PATH_INFO: `/v3/users/51edc434ecbf49b59ff6ea0b493929c0` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:18.205410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] RBAC: Authorizing `identity:delete_user(user_id=51edc434ecbf49b59ff6ea0b493929c0)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:18.207366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:18.349507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51edc434ecbf49b59ff6ea0b493929c0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.410628 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51edc434ecbf49b59ff6ea0b493929c0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.411465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51edc434ecbf49b59ff6ea0b493929c0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.419957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51edc434ecbf49b59ff6ea0b493929c0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.420604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e9cc74f7-1ca6-4fdd-b968-773401a9849c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51edc434ecbf49b59ff6ea0b493929c0'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:18.439553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1247/2521] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:18 2022] DELETE /identity/v3/users/51edc434ecbf49b59ff6ea0b493929c0 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:18.845826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:18.847629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:18.863405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035bc10f-a31c-4783-abd9-17ebb9568fa6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:18.922240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:18.923132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:18.923419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:18.923727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] PATH_INFO: `/v3/projects/58ed78ecfebf491090728db5dfe691b4` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:18.924914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=58ed78ecfebf491090728db5dfe691b4)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:18.926866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:18.942519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:19.067084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '58ed78ecfebf491090728db5dfe691b4', 'request_id': 'req-6b45912c-2081-4bd9-acf2-046b827388a7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:19.159047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '58ed78ecfebf491090728db5dfe691b4', 'request_id': 'req-6b45912c-2081-4bd9-acf2-046b827388a7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:19.165866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '58ed78ecfebf491090728db5dfe691b4', 'request_id': 'req-6b45912c-2081-4bd9-acf2-046b827388a7'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:19.177537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] The token cache is being invalidate because project 58ed78ecfebf491090728db5dfe691b4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:19.177830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:19.178485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6b45912c-2081-4bd9-acf2-046b827388a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:19.180184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1275/2522] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:18 2022] DELETE /identity/v3/projects/58ed78ecfebf491090728db5dfe691b4 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:19.185494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9640b-f8e8-4ff3-a1fa-c3398135c9cd None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.185977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9640b-f8e8-4ff3-a1fa-c3398135c9cd None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.186324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f9640b-f8e8-4ff3-a1fa-c3398135c9cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.234377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-44f9640b-f8e8-4ff3-a1fa-c3398135c9cd None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:19.235617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-44f9640b-f8e8-4ff3-a1fa-c3398135c9cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:19.293966 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1248/2523] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:19.301781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35fd5b7-e1d0-40db-b42c-b2244e41455f None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.302067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35fd5b7-e1d0-40db-b42c-b2244e41455f None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.302339 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35fd5b7-e1d0-40db-b42c-b2244e41455f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.320184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f35fd5b7-e1d0-40db-b42c-b2244e41455f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:19.321378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f35fd5b7-e1d0-40db-b42c-b2244e41455f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:19.329775 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1276/2524] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:19.333872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f9640b-f8e8-4ff3-a1fa-c3398135c9cd None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:19.340669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:19.344489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f9640b-f8e8-4ff3-a1fa-c3398135c9cd None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:19.349075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:19.352187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.352187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.352187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.352187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:26:19.352187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:19.353074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:19.366419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1249/2525] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:26:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:26:19.370873 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35fd5b7-e1d0-40db-b42c-b2244e41455f None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:19.380181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35fd5b7-e1d0-40db-b42c-b2244e41455f None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:19.385496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:19.386210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.386494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.386791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.387521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:19.389442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:19.451961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1277/2526] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:19 2022] POST /identity/v3/projects => generated 379 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:19.457381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:19.466675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199b6fab-ace7-43a5-86f2-2bb6d26846a1 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:19.524259 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:19.525587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.525867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.526141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.526866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:19.528802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:19.593598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1250/2527] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:19 2022] POST /identity/v3/users => generated 290 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:26:19.597589 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:19.607997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b5e212-b89d-4ded-9aa9-32fa6e133357 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:19.612573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:19.613245 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.613502 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.613774 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.614522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:19.616198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:19.627347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1278/2528] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:19 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:19.631248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:19.641795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1c2d56-e9cb-4e25-b71f-de4a21ed3285 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:19.646555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621662b9-b38d-4203-8b50-ff8e1d890097 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:19.647225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621662b9-b38d-4203-8b50-ff8e1d890097 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.647510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621662b9-b38d-4203-8b50-ff8e1d890097 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.647790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621662b9-b38d-4203-8b50-ff8e1d890097 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.648516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621662b9-b38d-4203-8b50-ff8e1d890097 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:19.650110 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621662b9-b38d-4203-8b50-ff8e1d890097 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:19.660076 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1251/2529] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:19 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:19.664135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:19.674198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217ce1d3-5cb3-4d45-9488-9379807094fc admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:19.678726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:19.679624 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.679904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.680197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] PATH_INFO: `/v3/projects/d2aca15269be4060a239ba6f4ca5f2e9/users/78eccf6d28d442a0b209e74d6cf70e30/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.694830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d2aca15269be4060a239ba6f4ca5f2e9, user_id=78eccf6d28d442a0b209e74d6cf70e30, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:19.698297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:19.787377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1279/2530] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:19 2022] PUT /identity/v3/projects/d2aca15269be4060a239ba6f4ca5f2e9/users/78eccf6d28d442a0b209e74d6cf70e30/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:19.851379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301cc913-7a15-4cbf-8cee-b369ba6ab33a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:19.851671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301cc913-7a15-4cbf-8cee-b369ba6ab33a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:19.851954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301cc913-7a15-4cbf-8cee-b369ba6ab33a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:19.880326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-301cc913-7a15-4cbf-8cee-b369ba6ab33a None None] MFA Rules not processed for user `78eccf6d28d442a0b209e74d6cf70e30`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:19.881575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-301cc913-7a15-4cbf-8cee-b369ba6ab33a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:19.948866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1252/2531] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:19 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:19.953391 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.013581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774ac13-2678-428b-bfae-903046f295f4 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.018091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.018739 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.019052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.019324 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.020007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.021898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.091284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1280/2532] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:19 2022] POST /identity/v3/projects => generated 379 bytes in 138 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:26:20.095927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301cc913-7a15-4cbf-8cee-b369ba6ab33a None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.106370 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301cc913-7a15-4cbf-8cee-b369ba6ab33a None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.161946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.162609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.162925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.163200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.163879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.165776 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.172505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:20.252557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1253/2533] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:20 2022] POST /identity/v3/users => generated 290 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:20.256689 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.268138 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03d70d7a-3af5-4917-a110-227c849dd76d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.272839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.273497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.273781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.274085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.275127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.277004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.288772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1281/2534] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:20 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:20.293584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.303488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d769d26-5309-4366-afaf-6b2c29311166 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.308783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7160079b-c0e4-496a-938c-13d86ec0b4ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.309439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7160079b-c0e4-496a-938c-13d86ec0b4ab admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.309723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7160079b-c0e4-496a-938c-13d86ec0b4ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.310011 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7160079b-c0e4-496a-938c-13d86ec0b4ab admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.310912 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7160079b-c0e4-496a-938c-13d86ec0b4ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.312508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7160079b-c0e4-496a-938c-13d86ec0b4ab admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.322634 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1254/2535] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:20 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:20.326642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.336427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63edad8b-a574-42f7-8fde-6273331cb7ed admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.341119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.344131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.344131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.344131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] PATH_INFO: `/v3/projects/f29cf6ef6d934e83afb72fdcec96e5ea/users/b716b54a0a72476a8a1f7defd38444db/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.353869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:20.357688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f29cf6ef6d934e83afb72fdcec96e5ea, user_id=b716b54a0a72476a8a1f7defd38444db, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.360367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.426246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1282/2536] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:20 2022] PUT /identity/v3/projects/f29cf6ef6d934e83afb72fdcec96e5ea/users/b716b54a0a72476a8a1f7defd38444db/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:20.437510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf7c7b-6413-4b38-906a-d0059c684e46 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.437803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf7c7b-6413-4b38-906a-d0059c684e46 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.438078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbf7c7b-6413-4b38-906a-d0059c684e46 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.465376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-afbf7c7b-6413-4b38-906a-d0059c684e46 None None] MFA Rules not processed for user `b716b54a0a72476a8a1f7defd38444db`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:20.466741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-afbf7c7b-6413-4b38-906a-d0059c684e46 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:20.531785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1255/2537] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:20 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:20.536144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.595535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f4caf75-ce1d-40c5-9784-fb456ce8d75c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.600104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.600753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.601031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.601302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.601979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.603918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.754842 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1283/2538] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:20 2022] POST /identity/v3/projects => generated 375 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:20.759889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf7c7b-6413-4b38-906a-d0059c684e46 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.770685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbf7c7b-6413-4b38-906a-d0059c684e46 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.827355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.827952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.828239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.828521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.829214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.831170 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.902265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1256/2539] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:20 2022] POST /identity/v3/users => generated 294 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:26:20.906194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.916573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35d2c6fc-5586-4a47-b19c-a56614d8f27b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.921627 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.922285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.922573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.922888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.923697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.925321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:20.936301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1284/2540] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:20.940304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:20.951038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbc651b-880d-45ac-afd6-5ed72d6f460c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:20.955922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:20.956797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:20.957086 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:20.957397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] PATH_INFO: `/v3/projects/ee1fa6191a444174aa52719f2ca1f771/users/8e8f0c50302c4d8d97eefec10febb6f2/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:20.971853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee1fa6191a444174aa52719f2ca1f771, user_id=8e8f0c50302c4d8d97eefec10febb6f2, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:20.974499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:21.033044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1257/2541] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:20 2022] PUT /identity/v3/projects/ee1fa6191a444174aa52719f2ca1f771/users/8e8f0c50302c4d8d97eefec10febb6f2/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:26:21.036971 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:21.049065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a624bf17-e939-4109-aaa1-6aca8f2eba02 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:21.106488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d26e00b-5f39-4731-b342-1767cef5708b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:21.107101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d26e00b-5f39-4731-b342-1767cef5708b admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:21.107381 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d26e00b-5f39-4731-b342-1767cef5708b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:21.107656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d26e00b-5f39-4731-b342-1767cef5708b admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:21.108372 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d26e00b-5f39-4731-b342-1767cef5708b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:21.110009 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d26e00b-5f39-4731-b342-1767cef5708b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:21.120161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1285/2542] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:21 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:21.124499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:21.135536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3827b625-401d-4714-8f0b-b24c44e2deab admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:21.140382 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:21.141269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:21.141569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:21.141851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] PATH_INFO: `/v3/projects/ee1fa6191a444174aa52719f2ca1f771/users/8e8f0c50302c4d8d97eefec10febb6f2/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:21.143902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee1fa6191a444174aa52719f2ca1f771, user_id=8e8f0c50302c4d8d97eefec10febb6f2, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:21.146664 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:21.157840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1258/2543] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:21 2022] PUT /identity/v3/projects/ee1fa6191a444174aa52719f2ca1f771/users/8e8f0c50302c4d8d97eefec10febb6f2/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:21.169353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:21.169645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:21.169928 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:21.195495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] MFA Rules not processed for user `8e8f0c50302c4d8d97eefec10febb6f2`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:21.197258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:21.242617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:21.263824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1286/2544] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:21 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:21.275439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:21.277188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:21.299665 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:21.334794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29d3187-ab86-4074-8d1e-b05ed3144204 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:21.339145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:21.339785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:21.340065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:21.340340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:21.349807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:21.351276 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:21.360792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1259/2545] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:21 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:21.469899 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:21.471887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:21.484219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed3eea8-8c83-4dac-948a-d9ca34052d16 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:21.487687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e8835e-253c-4a81-a780-e45a9cfa54e6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:21.488316 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e8835e-253c-4a81-a780-e45a9cfa54e6 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:21.488595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e8835e-253c-4a81-a780-e45a9cfa54e6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:21.488867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e8835e-253c-4a81-a780-e45a9cfa54e6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:21.544018 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e8835e-253c-4a81-a780-e45a9cfa54e6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:21.545457 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e8835e-253c-4a81-a780-e45a9cfa54e6 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:21.554206 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1287/2546] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:21 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:25.436498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:25.498807 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-222d2303-0394-4f2d-be22-bc0dc25b2091 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:25.503256 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:25.504118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:25.504398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:25.504672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] PATH_INFO: `/v3/users/78eccf6d28d442a0b209e74d6cf70e30` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:25.505855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] RBAC: Authorizing `identity:delete_user(user_id=78eccf6d28d442a0b209e74d6cf70e30)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:25.507841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:25.682371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '78eccf6d28d442a0b209e74d6cf70e30'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:25.742065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '78eccf6d28d442a0b209e74d6cf70e30'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:25.742869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '78eccf6d28d442a0b209e74d6cf70e30'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:25.750600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '78eccf6d28d442a0b209e74d6cf70e30'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:25.751217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '78eccf6d28d442a0b209e74d6cf70e30'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:25.769592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1260/2547] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:25 2022] DELETE /identity/v3/users/78eccf6d28d442a0b209e74d6cf70e30 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:25.775111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:25.775622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:25.775740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:25.792917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:25.798893 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:25.853965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1288/2548] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:25.863565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:25.864999 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:25.914013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:25.925772 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b8a865-6fb9-4020-b991-51a00c430609 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:25.929494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:25.930131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:25.930408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:25.931244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:25.940606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:25.942031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:25.953337 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1261/2549] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:26.371182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:26.372745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:26.386742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c4bed00-d369-4545-b578-5cc583887fb2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:26.391959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:26.392827 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:26.405890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:26.405890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] PATH_INFO: `/v3/projects/d2aca15269be4060a239ba6f4ca5f2e9` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:26.405890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=d2aca15269be4060a239ba6f4ca5f2e9)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:26.405890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:26.532471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd2aca15269be4060a239ba6f4ca5f2e9', 'request_id': 'req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.598834 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd2aca15269be4060a239ba6f4ca5f2e9', 'request_id': 'req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.605867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd2aca15269be4060a239ba6f4ca5f2e9', 'request_id': 'req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.617794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] The token cache is being invalidate because project d2aca15269be4060a239ba6f4ca5f2e9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:26.618038 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.618721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.620544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1289/2550] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:26 2022] DELETE /identity/v3/projects/d2aca15269be4060a239ba6f4ca5f2e9 => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:26.624642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:26.636257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92f8bcf-ce13-4cf2-8992-4aa9fcbf7829 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:26.702794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:26.703675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:26.703960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:26.704237 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] PATH_INFO: `/v3/users/b716b54a0a72476a8a1f7defd38444db` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:26.705426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] RBAC: Authorizing `identity:delete_user(user_id=b716b54a0a72476a8a1f7defd38444db)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:26.707451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:26.895610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b716b54a0a72476a8a1f7defd38444db'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.948418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b716b54a0a72476a8a1f7defd38444db'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.949032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b716b54a0a72476a8a1f7defd38444db'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.956652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b716b54a0a72476a8a1f7defd38444db'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.957344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b716b54a0a72476a8a1f7defd38444db'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:26.976200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1262/2551] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:26 2022] DELETE /identity/v3/users/b716b54a0a72476a8a1f7defd38444db => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:27.431910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:27.434054 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:27.449364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61a9e66f-0589-4a6e-ad47-16b3c1e868ee admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:27.454618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:27.510466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:27.511355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:27.511643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:27.511922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] PATH_INFO: `/v3/projects/f29cf6ef6d934e83afb72fdcec96e5ea` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:27.513155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f29cf6ef6d934e83afb72fdcec96e5ea)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:27.515203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:27.661988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f29cf6ef6d934e83afb72fdcec96e5ea', 'request_id': 'req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:27.747659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f29cf6ef6d934e83afb72fdcec96e5ea', 'request_id': 'req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:27.754807 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f29cf6ef6d934e83afb72fdcec96e5ea', 'request_id': 'req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:27.766201 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] The token cache is being invalidate because project f29cf6ef6d934e83afb72fdcec96e5ea was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:27.766492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:27.767175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:27.768852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1290/2552] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:27 2022] DELETE /identity/v3/projects/f29cf6ef6d934e83afb72fdcec96e5ea => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:27.772987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:27.774447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:27.788155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6681c0d8-3d21-42a5-8c12-fa498014c78a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:27.846064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:27.846967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:27.847251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:27.847527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] PATH_INFO: `/v3/users/8e8f0c50302c4d8d97eefec10febb6f2` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:27.848752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] RBAC: Authorizing `identity:delete_user(user_id=8e8f0c50302c4d8d97eefec10febb6f2)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:27.850741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:27.858140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:28.017643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8e8f0c50302c4d8d97eefec10febb6f2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.072560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8e8f0c50302c4d8d97eefec10febb6f2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.073333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8e8f0c50302c4d8d97eefec10febb6f2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.080650 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8e8f0c50302c4d8d97eefec10febb6f2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.081284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c2098f2-9769-460f-9ad2-584324a8406d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8e8f0c50302c4d8d97eefec10febb6f2'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.099519 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1263/2553] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:27 2022] DELETE /identity/v3/users/8e8f0c50302c4d8d97eefec10febb6f2 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:28.493345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:28.496393 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:28.509375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ded15ce-9ab8-41c0-9a62-7e41a436ae6d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:28.571742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:28.572521 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:28.573302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:28.573573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] PATH_INFO: `/v3/projects/ee1fa6191a444174aa52719f2ca1f771` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:28.574796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee1fa6191a444174aa52719f2ca1f771)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:28.576759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:28.649100 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:28.656552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee1fa6191a444174aa52719f2ca1f771', 'request_id': 'req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.718704 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee1fa6191a444174aa52719f2ca1f771', 'request_id': 'req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.725382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee1fa6191a444174aa52719f2ca1f771', 'request_id': 'req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.736272 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] The token cache is being invalidate because project ee1fa6191a444174aa52719f2ca1f771 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:28.736565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.737240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-ce1b0e23-7006-48eb-b072-d587cb1d1a2c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:28.738804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1291/2554] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:28 2022] DELETE /identity/v3/projects/ee1fa6191a444174aa52719f2ca1f771 => generated 0 bytes in 246 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:28.744842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5f274a-fc3d-409a-aebd-a2eba4d410da None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:28.745364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5f274a-fc3d-409a-aebd-a2eba4d410da None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:28.745861 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5f274a-fc3d-409a-aebd-a2eba4d410da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:28.764731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1a5f274a-fc3d-409a-aebd-a2eba4d410da None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:28.765924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1a5f274a-fc3d-409a-aebd-a2eba4d410da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:28.826109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1264/2555] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:28.834460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00627b1a-77f7-4141-aa41-d8c091510c09 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:28.834778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00627b1a-77f7-4141-aa41-d8c091510c09 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:28.835066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00627b1a-77f7-4141-aa41-d8c091510c09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:28.853985 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-00627b1a-77f7-4141-aa41-d8c091510c09 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:28.855262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-00627b1a-77f7-4141-aa41-d8c091510c09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:28.863420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1292/2556] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:28.867827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5f274a-fc3d-409a-aebd-a2eba4d410da None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:28.878305 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5f274a-fc3d-409a-aebd-a2eba4d410da None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:28.882860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:28.883518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:28.883802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:28.884079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:28.884833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:26:28.885142 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:28.887620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:28.899007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1265/2557] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:26:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:26:28.903529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00627b1a-77f7-4141-aa41-d8c091510c09 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:28.912865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00627b1a-77f7-4141-aa41-d8c091510c09 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:28.917772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:28.918409 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:28.918714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:28.919023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:28.919736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:28.921636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.018147 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1293/2558] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:28 2022] POST /identity/v3/projects => generated 379 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:29.022146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.032570 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09472c8-d92a-4017-9779-43c3fba0c307 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.089542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.090135 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.090420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.090698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.091490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.093543 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.215769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:29.220895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1266/2559] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:29 2022] POST /identity/v3/users => generated 290 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:29.224969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.235400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c869af-f46a-41c0-9dda-46ce55ef5907 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.239939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.240598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.240882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.241159 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.241875 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.243529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.254603 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1294/2560] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:29 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:29.261501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.271530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7c89602-a9a3-448a-8c43-7a832bfe22a7 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.276865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48a9ec16-2e08-4e12-bc92-9e9e95e3d478 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.277525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a9ec16-2e08-4e12-bc92-9e9e95e3d478 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.277808 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a9ec16-2e08-4e12-bc92-9e9e95e3d478 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.278088 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48a9ec16-2e08-4e12-bc92-9e9e95e3d478 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.278847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a9ec16-2e08-4e12-bc92-9e9e95e3d478 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.280454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48a9ec16-2e08-4e12-bc92-9e9e95e3d478 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.291022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1267/2561] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:29.295920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.305721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d037793-c626-43ed-a784-664978f3448c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.311252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.321965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.321965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.321965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] PATH_INFO: `/v3/projects/eb4d41cd20da47fabd54397cfa645735/users/ea39e209306c4978bddb17ce5838424f/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.327754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb4d41cd20da47fabd54397cfa645735, user_id=ea39e209306c4978bddb17ce5838424f, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.330525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.427558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1295/2562] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:29 2022] PUT /identity/v3/projects/eb4d41cd20da47fabd54397cfa645735/users/ea39e209306c4978bddb17ce5838424f/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:29.439344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737f364e-5bad-4b78-8d83-0c8f07d8ef65 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.439633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737f364e-5bad-4b78-8d83-0c8f07d8ef65 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.439910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-737f364e-5bad-4b78-8d83-0c8f07d8ef65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.465342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-737f364e-5bad-4b78-8d83-0c8f07d8ef65 None None] MFA Rules not processed for user `ea39e209306c4978bddb17ce5838424f`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:29.466523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-737f364e-5bad-4b78-8d83-0c8f07d8ef65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:29.529341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1268/2563] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:29 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:29.533786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.595347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fedb27ca-14db-4a98-91ab-4ab4b2a8e94b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.599973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.600625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.600942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.601217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.601908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.603838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.718134 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1296/2564] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:29 2022] POST /identity/v3/projects => generated 379 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:29.719263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737f364e-5bad-4b78-8d83-0c8f07d8ef65 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.731262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-737f364e-5bad-4b78-8d83-0c8f07d8ef65 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.787156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.787827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.788118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.788390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.789082 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.791053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.873923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1269/2565] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:29 2022] POST /identity/v3/users => generated 296 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:29.878001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.888398 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadde4e-8da1-4405-840a-5af2c733d352 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.893399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.894049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.894328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.894600 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.895497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.897085 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:29.908063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1297/2566] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:29 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:29.912210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:29.923618 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd75dd3b-55db-4d09-9766-d66a29684afa admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:29.930750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:29.931668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:29.931958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:29.932331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] PATH_INFO: `/v3/projects/8684a3483822445982786a499ad5e2c6/users/a7840bc6dfa44e96862269b88e47714b/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:29.949139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=8684a3483822445982786a499ad5e2c6, user_id=a7840bc6dfa44e96862269b88e47714b, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:29.952155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:30.027978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1270/2567] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:29 2022] PUT /identity/v3/projects/8684a3483822445982786a499ad5e2c6/users/a7840bc6dfa44e96862269b88e47714b/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 116 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:30.031792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:30.041785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faf6941b-039e-4f2a-9f31-46d192c87e34 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:30.075155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:30.097788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e8548f0-67af-471d-989a-2c7737a3e1c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:30.098459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8548f0-67af-471d-989a-2c7737a3e1c7 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:30.098818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8548f0-67af-471d-989a-2c7737a3e1c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:30.099029 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e8548f0-67af-471d-989a-2c7737a3e1c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:30.099749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8548f0-67af-471d-989a-2c7737a3e1c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:30.101334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e8548f0-67af-471d-989a-2c7737a3e1c7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:30.111126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1298/2568] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:30 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:30.115191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:30.125892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e89130e-547c-4a93-a7f9-924959bff7bb admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:30.130542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:30.131442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:30.131739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:30.132001 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] PATH_INFO: `/v3/projects/8684a3483822445982786a499ad5e2c6/users/a7840bc6dfa44e96862269b88e47714b/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:30.133968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8684a3483822445982786a499ad5e2c6, user_id=a7840bc6dfa44e96862269b88e47714b, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:30.137270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:30.147678 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1271/2569] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:30 2022] PUT /identity/v3/projects/8684a3483822445982786a499ad5e2c6/users/a7840bc6dfa44e96862269b88e47714b/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:30.159680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b2183-b1c7-45e3-b3f3-88c2207b6951 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:30.160043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b2183-b1c7-45e3-b3f3-88c2207b6951 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:30.160380 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd5b2183-b1c7-45e3-b3f3-88c2207b6951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:30.187923 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-fd5b2183-b1c7-45e3-b3f3-88c2207b6951 None None] MFA Rules not processed for user `a7840bc6dfa44e96862269b88e47714b`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:30.189166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fd5b2183-b1c7-45e3-b3f3-88c2207b6951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:30.256336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1299/2570] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:30 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:30.267866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:30.269521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:30.304197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:30.327723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e45bc44-d114-4493-ba05-c77077d1bbf4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:30.331484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bc8d58f-418c-40e8-ab39-b3de8733a84b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:30.332128 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc8d58f-418c-40e8-ab39-b3de8733a84b service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:30.332414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc8d58f-418c-40e8-ab39-b3de8733a84b service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:30.332695 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bc8d58f-418c-40e8-ab39-b3de8733a84b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:30.342670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc8d58f-418c-40e8-ab39-b3de8733a84b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:30.344161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bc8d58f-418c-40e8-ab39-b3de8733a84b service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:30.353698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1272/2571] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:30 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:30.841663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5b2183-b1c7-45e3-b3f3-88c2207b6951 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:30.903198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd5b2183-b1c7-45e3-b3f3-88c2207b6951 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:30.907597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:30.908451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:30.908727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:30.908994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] PATH_INFO: `/v3/users/ea39e209306c4978bddb17ce5838424f` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:30.910162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea39e209306c4978bddb17ce5838424f)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:30.912105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:31.091342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea39e209306c4978bddb17ce5838424f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:31.181837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea39e209306c4978bddb17ce5838424f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:31.187552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:31.189481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea39e209306c4978bddb17ce5838424f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:31.190121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea39e209306c4978bddb17ce5838424f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:31.190734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea39e209306c4978bddb17ce5838424f'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:31.207811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1300/2572] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:30 2022] DELETE /identity/v3/users/ea39e209306c4978bddb17ce5838424f => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:31.212365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:31.212892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:31.212996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:31.232134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:31.233338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:31.292967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1273/2573] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:31.302499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:31.303940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:31.385693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4931fb52-0f27-4f31-87e4-7e2d3f5311e6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:31.389188 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:31.389839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:31.390123 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:31.390397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:31.399357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:31.401413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:31.410867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1301/2574] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:31.868641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:31.868641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:31.876074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfb14bf-bf87-4c3d-aef1-564b59445300 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:31.880675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:31.881590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:31.881826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:31.882204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] PATH_INFO: `/v3/projects/eb4d41cd20da47fabd54397cfa645735` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:31.883327 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb4d41cd20da47fabd54397cfa645735)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:31.885801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:32.012354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb4d41cd20da47fabd54397cfa645735', 'request_id': 'req-16523020-170e-4052-aef2-5d6cdf0a8723'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.019732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb4d41cd20da47fabd54397cfa645735', 'request_id': 'req-16523020-170e-4052-aef2-5d6cdf0a8723'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.134054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb4d41cd20da47fabd54397cfa645735', 'request_id': 'req-16523020-170e-4052-aef2-5d6cdf0a8723'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.151865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] The token cache is being invalidate because project eb4d41cd20da47fabd54397cfa645735 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:32.151865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.151865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.151865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1274/2575] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:31 2022] DELETE /identity/v3/projects/eb4d41cd20da47fabd54397cfa645735 => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:32.153461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:32.163096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3108e56-8604-4999-baa7-0b5b2c7dad2c service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:32.221110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:32.221981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:32.222251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:32.222523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] PATH_INFO: `/v3/users/a7840bc6dfa44e96862269b88e47714b` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:32.223754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] RBAC: Authorizing `identity:delete_user(user_id=a7840bc6dfa44e96862269b88e47714b)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:32.225672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:32.441552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a7840bc6dfa44e96862269b88e47714b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.542257 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a7840bc6dfa44e96862269b88e47714b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.550044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a7840bc6dfa44e96862269b88e47714b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.550813 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a7840bc6dfa44e96862269b88e47714b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.551428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-7d667ca6-87b5-44cd-af5c-7196d9e6695e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a7840bc6dfa44e96862269b88e47714b'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:32.552574 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:32.569747 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1302/2576] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:26:32 2022] DELETE /identity/v3/users/a7840bc6dfa44e96862269b88e47714b => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:26:32.995214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:32.996729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:33.011092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16523020-170e-4052-aef2-5d6cdf0a8723 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:33.064452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:33.070581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:33.071471 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.071853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.072098 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] PATH_INFO: `/v3/projects/8684a3483822445982786a499ad5e2c6` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.073871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8684a3483822445982786a499ad5e2c6)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:33.075845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:33.192779 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8684a3483822445982786a499ad5e2c6', 'request_id': 'req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:33.198924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8684a3483822445982786a499ad5e2c6', 'request_id': 'req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:33.284869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8684a3483822445982786a499ad5e2c6', 'request_id': 'req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:33.297746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] The token cache is being invalidate because project 8684a3483822445982786a499ad5e2c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:26:33.298041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:33.298690 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bfe4dc96-b52a-49a5-b0ee-7a52a6dc4d32 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:26:33.300301 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1275/2577] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:26:32 2022] DELETE /identity/v3/projects/8684a3483822445982786a499ad5e2c6 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:26:33.305494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478422d1-c0f7-47f8-8e7e-3a1f671885b9 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.305960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478422d1-c0f7-47f8-8e7e-3a1f671885b9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.306055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478422d1-c0f7-47f8-8e7e-3a1f671885b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.325183 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-478422d1-c0f7-47f8-8e7e-3a1f671885b9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:33.326533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-478422d1-c0f7-47f8-8e7e-3a1f671885b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:33.396332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1303/2578] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:33.404246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcf0d12-d121-4862-946c-8f5895d0c82f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.408366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcf0d12-d121-4862-946c-8f5895d0c82f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.408366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcf0d12-d121-4862-946c-8f5895d0c82f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.423176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7fcf0d12-d121-4862-946c-8f5895d0c82f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:33.424373 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7fcf0d12-d121-4862-946c-8f5895d0c82f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:33.433821 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1276/2579] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:33.438103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478422d1-c0f7-47f8-8e7e-3a1f671885b9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:33.448791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478422d1-c0f7-47f8-8e7e-3a1f671885b9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:33.453584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:33.454263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.454569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.454877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.455612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:26:33.455922 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:33.457611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:33.469693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1304/2580] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:26:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 14 12:26:33.473939 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcf0d12-d121-4862-946c-8f5895d0c82f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:33.483390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcf0d12-d121-4862-946c-8f5895d0c82f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:33.488066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:33.488772 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.489051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.489326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.490025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:33.492064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:33.583330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1277/2581] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:26:33 2022] POST /identity/v3/projects => generated 363 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:33.587427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:33.598269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7156680f-790f-4930-ac9d-f522193175f6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:33.655829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:33.660795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.660795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.660795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.660795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:33.660795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:33.742604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1305/2582] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:26:33 2022] POST /identity/v3/users => generated 282 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:26:33.746936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:33.756982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2141139-a9e4-4158-be44-c58421ce70a2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:33.762277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:33.762959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.763239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.763513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.764232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:33.765828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:33.776869 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1278/2583] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:33.780963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:33.790963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-886c0232-bff5-4b36-8a09-3deb4cea7f3d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:33.796238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff6c711d-c104-4f48-85ff-e2ec718b7ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:33.796969 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c711d-c104-4f48-85ff-e2ec718b7ad4 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.798778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c711d-c104-4f48-85ff-e2ec718b7ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.798778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff6c711d-c104-4f48-85ff-e2ec718b7ad4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.798778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6c711d-c104-4f48-85ff-e2ec718b7ad4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:33.800099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff6c711d-c104-4f48-85ff-e2ec718b7ad4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:33.810414 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1306/2584] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:26:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:26:33.814572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:33.825249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36558024-78ff-460a-a5c1-28ff1aacf97d admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:33.830428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:33.831474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.837729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.837729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] PATH_INFO: `/v3/projects/77be0641ca5c4aa79dffa85dc6a2eaf2/users/e8b13ac1e95a4ae7bd7ad18694e9d92c/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.847841 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=77be0641ca5c4aa79dffa85dc6a2eaf2, user_id=e8b13ac1e95a4ae7bd7ad18694e9d92c, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:33.850499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:33.925059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1279/2585] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:26:33 2022] PUT /identity/v3/projects/77be0641ca5c4aa79dffa85dc6a2eaf2/users/e8b13ac1e95a4ae7bd7ad18694e9d92c/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:26:33.936732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ffb2e-6ecc-4ac6-976d-1183c0d0fcf8 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:33.937012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ffb2e-6ecc-4ac6-976d-1183c0d0fcf8 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:33.937290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95ffb2e-6ecc-4ac6-976d-1183c0d0fcf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:33.968686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-c95ffb2e-6ecc-4ac6-976d-1183c0d0fcf8 None None] MFA Rules not processed for user `e8b13ac1e95a4ae7bd7ad18694e9d92c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:33.970261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-c95ffb2e-6ecc-4ac6-976d-1183c0d0fcf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:34.009575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:34.039694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1307/2586] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:33 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:34.051787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:34.053564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:34.114629 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808a20f8-9657-428e-a932-34f3355b65f4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:34.118381 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:34.119062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:34.120349 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:34.120773 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:34.129152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:34.130589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:34.139920 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1280/2587] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:34 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:43.941370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:43.941903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:43.942003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:43.960110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] MFA Rules not processed for user `8d824401d1454a2bb19d7628e38b256a`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:26:43.961318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:44.016027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1308/2588] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:26:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:26:44.138666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=78,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:46.329974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:26:46.331507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:46.421716 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba316c64-986d-41df-a614-d757c252c489 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:26:46.426626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:26:46.427295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:26:46.427576 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:26:46.427850 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:26:46.429645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:26:46.442092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:26:46.443795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:26:46.454608 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1281/2589] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:26:46 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:26:54.008700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=36 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:26:56.450336 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=41 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:07.144777 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:07.146264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:07.202565 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97708ebc-bcb2-4350-819d-84829d97c4d7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:07.206808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:07.207471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:07.207759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:07.208041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:07.209903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:07.272373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:07.273812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:07.283828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1309/2590] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:11.036563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:11.038078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:11.054245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e55432d8-e358-41f7-8975-d4111a366e6d service placement] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:11.058775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:11.059666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:11.059951 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:11.060261 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] PATH_INFO: `/v3/users/15e708082e294dd9a0834ef21ea46fd9` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:11.061481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] RBAC: Authorizing `identity:delete_user(user_id=15e708082e294dd9a0834ef21ea46fd9)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:11.063721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:11.239715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '15e708082e294dd9a0834ef21ea46fd9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.317761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '15e708082e294dd9a0834ef21ea46fd9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.318333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '15e708082e294dd9a0834ef21ea46fd9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.325902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '15e708082e294dd9a0834ef21ea46fd9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.326544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-6cdc7657-5642-4280-8622-7eac661c6110 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '15e708082e294dd9a0834ef21ea46fd9'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.343945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1282/2591] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:11 2022] DELETE /identity/v3/users/15e708082e294dd9a0834ef21ea46fd9 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:11.616701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:11.628733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f32aabb2-c107-4a36-8d44-3c9e44de4acf service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:11.691156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:11.692076 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:11.692355 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:11.692628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] PATH_INFO: `/v3/projects/51617cc935204a6cba46c6d4d6a24cb3` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:11.693835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] RBAC: Authorizing `identity:delete_project(project_id=51617cc935204a6cba46c6d4d6a24cb3)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:11.695986 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:11.774943 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '51617cc935204a6cba46c6d4d6a24cb3', 'request_id': 'req-a24ff6d6-1bba-4371-bbc4-9ad292c44369'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.777450 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:11.871868 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '51617cc935204a6cba46c6d4d6a24cb3', 'request_id': 'req-a24ff6d6-1bba-4371-bbc4-9ad292c44369'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.879203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '51617cc935204a6cba46c6d4d6a24cb3', 'request_id': 'req-a24ff6d6-1bba-4371-bbc4-9ad292c44369'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.890744 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] The token cache is being invalidate because project 51617cc935204a6cba46c6d4d6a24cb3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:11.891062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.891702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a24ff6d6-1bba-4371-bbc4-9ad292c44369 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:11.894098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1310/2592] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:11 2022] DELETE /identity/v3/projects/51617cc935204a6cba46c6d4d6a24cb3 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:11.899411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc5f7cc-d86e-462a-9f64-7584b9aa0d00 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:11.899702 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc5f7cc-d86e-462a-9f64-7584b9aa0d00 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:11.899979 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc5f7cc-d86e-462a-9f64-7584b9aa0d00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:11.917715 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-9bc5f7cc-d86e-462a-9f64-7584b9aa0d00 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:11.919017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9bc5f7cc-d86e-462a-9f64-7584b9aa0d00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:11.979289 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1283/2593] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:11.988034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965562b9-ebb9-4b25-962e-5e4203161477 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:11.988314 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965562b9-ebb9-4b25-962e-5e4203161477 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:11.988626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-965562b9-ebb9-4b25-962e-5e4203161477 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.007053 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-965562b9-ebb9-4b25-962e-5e4203161477 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:12.008258 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-965562b9-ebb9-4b25-962e-5e4203161477 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:12.016399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1311/2594] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:12.020686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc5f7cc-d86e-462a-9f64-7584b9aa0d00 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.030245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc5f7cc-d86e-462a-9f64-7584b9aa0d00 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.035396 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.036047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.036330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.036606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.037331 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:27:12.037641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.039299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:12.051711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1284/2595] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:27:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:27:12.055961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965562b9-ebb9-4b25-962e-5e4203161477 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.065864 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-965562b9-ebb9-4b25-962e-5e4203161477 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.070680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.071408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.071722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.072016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.072746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.074733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:12.171699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1312/2596] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:12 2022] POST /identity/v3/projects => generated 403 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:12.176485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.187299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1729a5b3-1b22-4af9-9275-aa09c0506961 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.247103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.247775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.248062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.248345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.249074 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.251131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:12.257897 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:12.329604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1285/2597] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:12 2022] POST /identity/v3/users => generated 302 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:12.334352 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.344614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6224a665-1414-48c2-b957-857a3b4a1f6e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.350160 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.350902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.351187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.351464 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.352189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.353797 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:12.364428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1313/2598] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:12 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:12.368330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.379347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ea83ce8-8835-456e-bf1d-0808a9862c10 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.384049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c703fc57-b2e9-4095-9198-4d5d2d5ee21c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.384755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703fc57-b2e9-4095-9198-4d5d2d5ee21c admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.385044 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703fc57-b2e9-4095-9198-4d5d2d5ee21c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.385353 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c703fc57-b2e9-4095-9198-4d5d2d5ee21c admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.386211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c703fc57-b2e9-4095-9198-4d5d2d5ee21c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.387981 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c703fc57-b2e9-4095-9198-4d5d2d5ee21c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:12.401679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1286/2599] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:12 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:12.406083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.419968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19483b4-f214-443f-8648-46c69804a012 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.426195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.427102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.427415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.427782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] PATH_INFO: `/v3/projects/4be676ec9f124081afa4f4406e5c4218/users/8c02fc98489844f69c5dcc06805ab55a/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.444347 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4be676ec9f124081afa4f4406e5c4218, user_id=8c02fc98489844f69c5dcc06805ab55a, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.447149 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:12.557526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1314/2600] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:12 2022] PUT /identity/v3/projects/4be676ec9f124081afa4f4406e5c4218/users/8c02fc98489844f69c5dcc06805ab55a/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:12.569833 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcaade6-5017-4392-a43c-88d2072e2eb2 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.570127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcaade6-5017-4392-a43c-88d2072e2eb2 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.570406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dcaade6-5017-4392-a43c-88d2072e2eb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.597205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7dcaade6-5017-4392-a43c-88d2072e2eb2 None None] MFA Rules not processed for user `8c02fc98489844f69c5dcc06805ab55a`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:12.598449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7dcaade6-5017-4392-a43c-88d2072e2eb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:12.663598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1287/2601] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:12 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:12.672203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.732523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2532fdb-2a9d-4a15-b881-83a1f6a55657 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.738174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.738910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.739219 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.739490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.740181 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.742073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:12.867072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1315/2602] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:12 2022] POST /identity/v3/projects => generated 405 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:12.871153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcaade6-5017-4392-a43c-88d2072e2eb2 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:12.884867 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dcaade6-5017-4392-a43c-88d2072e2eb2 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:12.947038 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:12.947714 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:12.947998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:12.948275 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:12.948965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:12.950908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:13.053047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1288/2603] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:12 2022] POST /identity/v3/users => generated 309 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:13.057036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:13.067407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f2837f-9d33-426d-a6c9-84ed7ea4e243 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:13.072208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:13.072871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.073156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.073438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.074163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:13.075889 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:13.087428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1316/2604] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:13 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:13.091719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:13.102425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c531a526-aa52-435b-b2f1-2292b01086fd admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:13.107092 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:13.107974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.108257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.108541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] PATH_INFO: `/v3/projects/ac6245e27f784244a8b3e4a865c14a68/users/a3968f8e074d494abb685b8b0aef36c5/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.123310 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac6245e27f784244a8b3e4a865c14a68, user_id=a3968f8e074d494abb685b8b0aef36c5, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:13.125949 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:13.191620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1289/2605] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:13 2022] PUT /identity/v3/projects/ac6245e27f784244a8b3e4a865c14a68/users/a3968f8e074d494abb685b8b0aef36c5/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:13.195715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:13.205882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b407b3-54cb-479e-b692-ba90b31edce5 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:13.253501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:13.263378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bd00821-e162-4e19-8277-0158432219c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:13.264046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd00821-e162-4e19-8277-0158432219c3 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.264332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd00821-e162-4e19-8277-0158432219c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.264612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bd00821-e162-4e19-8277-0158432219c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.265335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd00821-e162-4e19-8277-0158432219c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:13.266980 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bd00821-e162-4e19-8277-0158432219c3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:13.277133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1317/2606] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:13 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:13.284187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:13.292520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd9b30ad-3b1a-413f-add6-86ffc56cdde6 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:13.297138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:13.298077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.298362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.298639 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] PATH_INFO: `/v3/projects/ac6245e27f784244a8b3e4a865c14a68/users/a3968f8e074d494abb685b8b0aef36c5/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.300667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ac6245e27f784244a8b3e4a865c14a68, user_id=a3968f8e074d494abb685b8b0aef36c5, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:13.303326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:13.313842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1290/2607] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:13 2022] PUT /identity/v3/projects/ac6245e27f784244a8b3e4a865c14a68/users/a3968f8e074d494abb685b8b0aef36c5/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:13.324674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c802c9-42ea-45db-b8ce-7a8e5d264af7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.330784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c802c9-42ea-45db-b8ce-7a8e5d264af7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.330784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c802c9-42ea-45db-b8ce-7a8e5d264af7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.352802 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-72c802c9-42ea-45db-b8ce-7a8e5d264af7 None None] MFA Rules not processed for user `a3968f8e074d494abb685b8b0aef36c5`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:13.354016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-72c802c9-42ea-45db-b8ce-7a8e5d264af7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:13.423323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1318/2608] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:13 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:13.433994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:13.436463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:13.460419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:INSERT=8,keystone:SELECT=92 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:13.516394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f871ff74-023b-4a17-b2b8-706fdcaf6f32 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:13.521494 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:13.522220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.522447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.522732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.532105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:13.533558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:13.543325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1291/2609] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:13 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:13.690394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c802c9-42ea-45db-b8ce-7a8e5d264af7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:13.754377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c802c9-42ea-45db-b8ce-7a8e5d264af7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:13.758865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:13.759734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.760023 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.760302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] PATH_INFO: `/v3/users/8c02fc98489844f69c5dcc06805ab55a` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.761492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c02fc98489844f69c5dcc06805ab55a)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:13.763471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:13.948470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:13.949914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:13.963843 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a693f9d6-f9d8-46d7-8a85-1742f8634333 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:13.969394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:13.970410 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:13.970694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:13.970997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] PATH_INFO: `/v3/users/e8b13ac1e95a4ae7bd7ad18694e9d92c` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:13.972082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c02fc98489844f69c5dcc06805ab55a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:13.972283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8b13ac1e95a4ae7bd7ad18694e9d92c)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:13.974605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:14.054546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c02fc98489844f69c5dcc06805ab55a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.062225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c02fc98489844f69c5dcc06805ab55a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.062908 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c02fc98489844f69c5dcc06805ab55a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.063525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-611075e2-4185-4df2-8b6b-dea68a7be6da admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c02fc98489844f69c5dcc06805ab55a'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.071596 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:14.081284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1319/2610] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:13 2022] DELETE /identity/v3/users/8c02fc98489844f69c5dcc06805ab55a => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:14.085812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:14.086099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:14.086377 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:14.103823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:14.105090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.144355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8b13ac1e95a4ae7bd7ad18694e9d92c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.163772 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1320/2611] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:27:14.173647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:14.174973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.220940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8b13ac1e95a4ae7bd7ad18694e9d92c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.221624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8b13ac1e95a4ae7bd7ad18694e9d92c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.230140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8b13ac1e95a4ae7bd7ad18694e9d92c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.230973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e1d82218-4921-4aa3-a836-2a3579fb84ca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8b13ac1e95a4ae7bd7ad18694e9d92c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.231544 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c143ca3-2e23-4490-a472-564a5716945b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:14.235354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:14.236062 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:14.236349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:14.236626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:14.239331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.249685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1292/2612] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:13 2022] DELETE /identity/v3/users/e8b13ac1e95a4ae7bd7ad18694e9d92c => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:27:14.251294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:14.252738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:14.254538 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:14.254864 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:14.255152 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:14.262865 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1321/2613] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:27:14.276633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:14.277847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.340200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1293/2614] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:27:14.351419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:14.353884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.364843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-744de64e-ee06-4108-bd35-85cefce2ce1f service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:14.414047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:14.414694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:14.415012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:14.415290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:14.426485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:14.428025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:14.438369 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1322/2615] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:14.670124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:14.671672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.686277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481cf74f-929b-499c-b7e0-fce652391b6e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:14.692454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:14.693201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:14.693497 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:14.694346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] PATH_INFO: `/v3/projects/4be676ec9f124081afa4f4406e5c4218` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:14.695670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4be676ec9f124081afa4f4406e5c4218)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:14.697968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:14.719242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:14.804147 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4be676ec9f124081afa4f4406e5c4218', 'request_id': 'req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.810273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4be676ec9f124081afa4f4406e5c4218', 'request_id': 'req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.845376 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:14.846955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.857569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4be676ec9f124081afa4f4406e5c4218', 'request_id': 'req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.862126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0424b0a3-be43-4c86-85b4-0e4f02928ecf service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:14.877954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] The token cache is being invalidate because project 4be676ec9f124081afa4f4406e5c4218 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:14.878810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.879565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:14.881328 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1294/2616] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:14 2022] DELETE /identity/v3/projects/4be676ec9f124081afa4f4406e5c4218 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:14.885071 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:14.886470 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:14.904189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:14.927680 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:14.928542 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:14.928830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:14.929105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] PATH_INFO: `/v3/projects/77be0641ca5c4aa79dffa85dc6a2eaf2` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:14.930309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77be0641ca5c4aa79dffa85dc6a2eaf2)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:14.932287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:14.959635 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3847fa02-f7e9-48c5-8c9a-5dced2cedc6c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:14.966440 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:14.967405 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:14.967682 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:14.967959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] PATH_INFO: `/v3/users/a3968f8e074d494abb685b8b0aef36c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:14.969217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=a3968f8e074d494abb685b8b0aef36c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:14.971299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.058007 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77be0641ca5c4aa79dffa85dc6a2eaf2', 'request_id': 'req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.132644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77be0641ca5c4aa79dffa85dc6a2eaf2', 'request_id': 'req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.139361 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77be0641ca5c4aa79dffa85dc6a2eaf2', 'request_id': 'req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.147052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a3968f8e074d494abb685b8b0aef36c5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.150671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] The token cache is being invalidate because project 77be0641ca5c4aa79dffa85dc6a2eaf2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:15.150989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.151635 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-1e0ea426-7ffe-4b83-8a0c-8497bfe40b61 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.153182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1323/2617] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:14 2022] DELETE /identity/v3/projects/77be0641ca5c4aa79dffa85dc6a2eaf2 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:27:15.158232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567609ad-31a9-4f2f-9936-e4b842e5308b None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.158519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567609ad-31a9-4f2f-9936-e4b842e5308b None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.158852 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-567609ad-31a9-4f2f-9936-e4b842e5308b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.177622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-567609ad-31a9-4f2f-9936-e4b842e5308b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:15.178859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-567609ad-31a9-4f2f-9936-e4b842e5308b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:15.203106 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a3968f8e074d494abb685b8b0aef36c5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.203761 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a3968f8e074d494abb685b8b0aef36c5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.211400 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a3968f8e074d494abb685b8b0aef36c5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.212075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e6ea7b49-ba39-4b61-bd8a-04c56a9a2dbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a3968f8e074d494abb685b8b0aef36c5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.229667 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1295/2618] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:14 2022] DELETE /identity/v3/users/a3968f8e074d494abb685b8b0aef36c5 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:27:15.238578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1324/2619] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:27:15.246671 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfd838-6280-4e5d-8eab-60d1469725cb None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.247302 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfd838-6280-4e5d-8eab-60d1469725cb None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.247536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10bfd838-6280-4e5d-8eab-60d1469725cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.268752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-10bfd838-6280-4e5d-8eab-60d1469725cb None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:15.269963 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-10bfd838-6280-4e5d-8eab-60d1469725cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:15.282721 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:15.331689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1296/2620] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:15.335905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567609ad-31a9-4f2f-9936-e4b842e5308b None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:15.345832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-567609ad-31a9-4f2f-9936-e4b842e5308b None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:15.351199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:15.351866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.352158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.352438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.353220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:27:15.353532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:15.355221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.367529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1325/2621] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:27:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:27:15.371750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfd838-6280-4e5d-8eab-60d1469725cb None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:15.381677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10bfd838-6280-4e5d-8eab-60d1469725cb None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:15.386248 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:15.386930 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.387233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.387493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.388215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:15.390133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.489787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1297/2622] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:15 2022] POST /identity/v3/projects => generated 391 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:15.494073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:15.505859 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25cd9d69-018e-48ca-bb2a-7cc26913df83 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:15.558805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:15.565719 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:15.566458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.566743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.567050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.567740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:15.569702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.672617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1326/2623] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:15 2022] POST /identity/v3/users => generated 296 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:15.680445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:15.680445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:15.683277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:15.690030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0da7d1eb-5065-4f16-b9c7-6c5501180490 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:15.695196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9841f36f-1d95-4f5a-a8af-17d7c467d645 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:15.696573 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:15.698083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.699189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.703935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] PATH_INFO: `/v3/projects/ac6245e27f784244a8b3e4a865c14a68` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.703935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] RBAC: Authorizing `identity:delete_project(project_id=ac6245e27f784244a8b3e4a865c14a68)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:15.704647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:15.704647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.704647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.705139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.705239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.705239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:15.705426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.717437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1327/2624] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:15.721280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:15.738437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda6ae67-16e8-4d46-b7e7-0c034be92324 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:15.738437 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:15.739027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.739341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.739615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.740344 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:15.741940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.752046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1328/2625] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:27:15.756098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:15.765686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab8a039-57a8-496a-9e70-5e60f7c12b6f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:15.770513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-226e887a-3c78-457a-b2c5-2fb3a6828e2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:15.771411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e887a-3c78-457a-b2c5-2fb3a6828e2f admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.771721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e887a-3c78-457a-b2c5-2fb3a6828e2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.771998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-226e887a-3c78-457a-b2c5-2fb3a6828e2f admin admin] PATH_INFO: `/v3/projects/b5c11acfe02f4346aa5d6ecc25a5d07c/users/0d1fdf3c170249eebf03b8a29b4f47e3/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.787436 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226e887a-3c78-457a-b2c5-2fb3a6828e2f admin admin] RBAC: Authorizing `identity:create_grant(project_id=b5c11acfe02f4346aa5d6ecc25a5d07c, user_id=0d1fdf3c170249eebf03b8a29b4f47e3, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:15.790202 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-226e887a-3c78-457a-b2c5-2fb3a6828e2f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:15.893295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1329/2626] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:15 2022] PUT /identity/v3/projects/b5c11acfe02f4346aa5d6ecc25a5d07c/users/0d1fdf3c170249eebf03b8a29b4f47e3/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 138 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:27:15.903844 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:15.904137 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:15.904420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:15.906537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ac6245e27f784244a8b3e4a865c14a68', 'request_id': 'req-17a6e659-e9bc-4745-b0cc-47dab9d09aae'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.913061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ac6245e27f784244a8b3e4a865c14a68', 'request_id': 'req-17a6e659-e9bc-4745-b0cc-47dab9d09aae'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.933302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] MFA Rules not processed for user `0d1fdf3c170249eebf03b8a29b4f47e3`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:15.934475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:15.991531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ac6245e27f784244a8b3e4a865c14a68', 'request_id': 'req-17a6e659-e9bc-4745-b0cc-47dab9d09aae'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:15.999485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1330/2627] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:15 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:16.007860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] The token cache is being invalidate because project ac6245e27f784244a8b3e4a865c14a68 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:16.008217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:16.008956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-17a6e659-e9bc-4745-b0cc-47dab9d09aae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:16.010288 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.010855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1298/2628] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:15 2022] DELETE /identity/v3/projects/ac6245e27f784244a8b3e4a865c14a68 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:16.011954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:16.015399 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcacf01-452c-46fb-a600-b662910eda0a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.015735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcacf01-452c-46fb-a600-b662910eda0a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.016017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1dcacf01-452c-46fb-a600-b662910eda0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.035233 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-1dcacf01-452c-46fb-a600-b662910eda0a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:16.036589 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-1dcacf01-452c-46fb-a600-b662910eda0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:16.070177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad14272b-d565-4966-9ece-2afa3b575d35 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.073811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.074458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.074742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.075066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.076861 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:16.088837 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.090280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.096737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1299/2629] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:16.100030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1331/2630] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:16 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:27:16.104953 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c077143e-aee9-494e-b0b5-d4795a2c60c7 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.105198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c077143e-aee9-494e-b0b5-d4795a2c60c7 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.105479 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c077143e-aee9-494e-b0b5-d4795a2c60c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.124009 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c077143e-aee9-494e-b0b5-d4795a2c60c7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:16.125216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c077143e-aee9-494e-b0b5-d4795a2c60c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:16.133528 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1300/2631] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:16.137523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.148913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e9d5bd-3cb6-4a9d-91af-2011c959c525 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.153734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.154448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.154733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.155064 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.155792 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:27:16.156104 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.157746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.170463 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1332/2632] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:27:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:27:16.174596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c077143e-aee9-494e-b0b5-d4795a2c60c7 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.184376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c077143e-aee9-494e-b0b5-d4795a2c60c7 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.189161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.189899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.190251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.190643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.191647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.193620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.203649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:16.329015 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1301/2633] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:16 2022] POST /identity/v3/projects => generated 365 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:16.330107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.340832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a7f011a-d442-42c3-be12-1fa7672571d4 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.353114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:16.397186 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.397876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.398158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.398435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.399162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.401158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.492679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1333/2634] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:16 2022] POST /identity/v3/users => generated 283 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:27:16.496973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.508265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc0e7d1-9023-462c-b3bd-42f6c5b2afc4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.513379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.514123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.514455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.514751 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.515546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.517354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.529503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1302/2635] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:16.533617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.543916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72edffaa-f7a0-481a-a79f-94dcbf2c1b81 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.548676 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c13e86b-17f5-4b05-b537-7ed42dd1287a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.551938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13e86b-17f5-4b05-b537-7ed42dd1287a admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.551938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13e86b-17f5-4b05-b537-7ed42dd1287a admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.551938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c13e86b-17f5-4b05-b537-7ed42dd1287a admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.551938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c13e86b-17f5-4b05-b537-7ed42dd1287a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.552422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c13e86b-17f5-4b05-b537-7ed42dd1287a admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.562424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1334/2636] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:16 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:16.567171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.577500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ff2517-2682-4079-91d6-639e697d1556 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.582754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.590919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.590919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.590919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] PATH_INFO: `/v3/projects/fa4b563611ad4933a98c0a32f1e9374a/users/b7d11107de584a9f96009309a33ef095/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.598957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa4b563611ad4933a98c0a32f1e9374a, user_id=b7d11107de584a9f96009309a33ef095, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.601687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.707989 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1303/2637] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:16 2022] PUT /identity/v3/projects/fa4b563611ad4933a98c0a32f1e9374a/users/b7d11107de584a9f96009309a33ef095/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:16.720143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e5a4dd-c0b6-4b84-896a-29c29c6f8342 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.720558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e5a4dd-c0b6-4b84-896a-29c29c6f8342 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.721044 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e5a4dd-c0b6-4b84-896a-29c29c6f8342 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.754350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-24e5a4dd-c0b6-4b84-896a-29c29c6f8342 None None] MFA Rules not processed for user `b7d11107de584a9f96009309a33ef095`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:16.757312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-24e5a4dd-c0b6-4b84-896a-29c29c6f8342 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:16.822851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1335/2638] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:16 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:16.832670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:16.896150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97a795a1-2bce-4c95-b3e4-b7b9270c3d53 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:16.900896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:16.901593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:16.901879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:16.902272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:16.903105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:16.905081 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:16.998645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1304/2639] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:16 2022] POST /identity/v3/projects => generated 365 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:17.002872 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e5a4dd-c0b6-4b84-896a-29c29c6f8342 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:17.014393 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e5a4dd-c0b6-4b84-896a-29c29c6f8342 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:17.071363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:17.072028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.072335 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.072614 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.073299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:17.075232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:17.163485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1336/2640] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:17 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:17.167752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:17.178977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7870f04a-4ab6-41b8-96c0-48905a54d464 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:17.183606 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:17.184262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.184548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.184829 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.185565 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:17.187202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:17.197600 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1305/2641] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:17.201486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:17.212427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ecccb9-3b8d-40b3-b713-6fc9b4a40822 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:17.217273 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:17.218168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.218425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.218694 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] PATH_INFO: `/v3/projects/43cc64fc73814015b20cd092c41efdef/users/93dfac72e6c94fe0b1f8b9a55ab5b3a8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.231311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:17.234256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=43cc64fc73814015b20cd092c41efdef, user_id=93dfac72e6c94fe0b1f8b9a55ab5b3a8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:17.236942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:17.306240 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1337/2642] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:17 2022] PUT /identity/v3/projects/43cc64fc73814015b20cd092c41efdef/users/93dfac72e6c94fe0b1f8b9a55ab5b3a8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:17.310307 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:17.320786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3319f6-8de9-4e4d-85af-0881f49d2dd8 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:17.378503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b46073-2aa0-4b44-a4a1-f88a515e8ebf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:17.379207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b46073-2aa0-4b44-a4a1-f88a515e8ebf admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.379554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b46073-2aa0-4b44-a4a1-f88a515e8ebf admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.379797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b46073-2aa0-4b44-a4a1-f88a515e8ebf admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.380574 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b46073-2aa0-4b44-a4a1-f88a515e8ebf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:17.382186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b46073-2aa0-4b44-a4a1-f88a515e8ebf admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:17.391992 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1306/2643] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:17 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:17.396851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:17.407040 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6de0b0b-2aaa-4eb3-8918-b3751419ec15 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:17.412599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:17.413187 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.413471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.413746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] PATH_INFO: `/v3/projects/43cc64fc73814015b20cd092c41efdef/users/93dfac72e6c94fe0b1f8b9a55ab5b3a8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.415869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] RBAC: Authorizing `identity:create_grant(project_id=43cc64fc73814015b20cd092c41efdef, user_id=93dfac72e6c94fe0b1f8b9a55ab5b3a8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:17.418882 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:17.429768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1338/2644] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:17 2022] PUT /identity/v3/projects/43cc64fc73814015b20cd092c41efdef/users/93dfac72e6c94fe0b1f8b9a55ab5b3a8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:17.440068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.440354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.440709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.466991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] MFA Rules not processed for user `93dfac72e6c94fe0b1f8b9a55ab5b3a8`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:17.468189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:17.480965 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:17.534204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1307/2645] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:17 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:17.546458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:17.548243 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:17.606354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a081a6f1-8378-469a-a8c5-d89a7a3244da admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:17.609933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:17.610569 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.610890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.611169 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.620784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:17.622327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:17.632357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1339/2646] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:17 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:17.877225 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:17.879533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:17.891138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bb207e0-0693-41a1-8215-deea1fc12178 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:17.896149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74671877-3f00-45b2-b938-10e536f64005 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:17.896802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74671877-3f00-45b2-b938-10e536f64005 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:17.897075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74671877-3f00-45b2-b938-10e536f64005 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:17.897347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74671877-3f00-45b2-b938-10e536f64005 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:17.954004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74671877-3f00-45b2-b938-10e536f64005 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:17.955546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74671877-3f00-45b2-b938-10e536f64005 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:17.965162 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1308/2647] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:17 2022] GET /identity/v3/auth/tokens => generated 3232 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:22.358128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:22.423582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d867db8-3ddc-4119-8b4d-dc8efe0cbaf7 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:22.428471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:22.429336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:22.429618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:22.429897 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] PATH_INFO: `/v3/users/b7d11107de584a9f96009309a33ef095` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:22.431118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7d11107de584a9f96009309a33ef095)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:22.433050 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:22.703096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7d11107de584a9f96009309a33ef095'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:22.791776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7d11107de584a9f96009309a33ef095'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:22.799498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7d11107de584a9f96009309a33ef095'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:22.800143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7d11107de584a9f96009309a33ef095'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:22.800758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7d11107de584a9f96009309a33ef095'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:22.815280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:22.820400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1340/2648] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:22 2022] DELETE /identity/v3/users/b7d11107de584a9f96009309a33ef095 => generated 0 bytes in 463 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:27:22.824997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:22.825511 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:22.825615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:22.842705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:22.847827 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:22.912078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1309/2649] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:27:22.921822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:22.923241 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:22.981180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0ae39e-2afd-4fda-acfb-0bcae292fe4d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:22.985049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:22.985730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:22.986020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:22.986296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:22.996478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:22.998011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:23.007835 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1341/2650] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:23.425285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:23.426710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:23.438968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd1bce8-4dc5-4e8e-ab13-a7daaa5c718e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:23.443950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:23.444958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:23.445252 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:23.445564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] PATH_INFO: `/v3/projects/fa4b563611ad4933a98c0a32f1e9374a` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:23.446924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa4b563611ad4933a98c0a32f1e9374a)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:23.448891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:23.548238 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:23.550974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa4b563611ad4933a98c0a32f1e9374a', 'request_id': 'req-e8cea31c-705d-47eb-8f94-de24df2048ae'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:23.556806 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa4b563611ad4933a98c0a32f1e9374a', 'request_id': 'req-e8cea31c-705d-47eb-8f94-de24df2048ae'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:23.640546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa4b563611ad4933a98c0a32f1e9374a', 'request_id': 'req-e8cea31c-705d-47eb-8f94-de24df2048ae'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:23.652058 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] The token cache is being invalidate because project fa4b563611ad4933a98c0a32f1e9374a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:23.652340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:23.652967 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:23.654487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1310/2651] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:23 2022] DELETE /identity/v3/projects/fa4b563611ad4933a98c0a32f1e9374a => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:23.658484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:23.669294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15f93e0a-6e90-431a-8d18-744062ab767c service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:23.726858 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:23.727732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:23.728014 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:23.728291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] PATH_INFO: `/v3/users/93dfac72e6c94fe0b1f8b9a55ab5b3a8` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:23.729497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=93dfac72e6c94fe0b1f8b9a55ab5b3a8)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:23.731481 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:23.931830 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '93dfac72e6c94fe0b1f8b9a55ab5b3a8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.031715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '93dfac72e6c94fe0b1f8b9a55ab5b3a8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.039266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '93dfac72e6c94fe0b1f8b9a55ab5b3a8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.039888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '93dfac72e6c94fe0b1f8b9a55ab5b3a8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.040489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5bba47a4-f6a9-4a98-8e0d-bf7837a5b3a1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '93dfac72e6c94fe0b1f8b9a55ab5b3a8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.057556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1342/2652] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:23 2022] DELETE /identity/v3/users/93dfac72e6c94fe0b1f8b9a55ab5b3a8 => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:24.462346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:24.463853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:24.478006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cea31c-705d-47eb-8f94-de24df2048ae admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:24.536558 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:24.538041 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:24.538393 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:24.538681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] PATH_INFO: `/v3/projects/43cc64fc73814015b20cd092c41efdef` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:24.539910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] RBAC: Authorizing `identity:delete_project(project_id=43cc64fc73814015b20cd092c41efdef)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:24.541840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:24.658123 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '43cc64fc73814015b20cd092c41efdef', 'request_id': 'req-1538a86a-b498-4164-a55d-b9ff5e6d979d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.664354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '43cc64fc73814015b20cd092c41efdef', 'request_id': 'req-1538a86a-b498-4164-a55d-b9ff5e6d979d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.720976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '43cc64fc73814015b20cd092c41efdef', 'request_id': 'req-1538a86a-b498-4164-a55d-b9ff5e6d979d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.733977 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] The token cache is being invalidate because project 43cc64fc73814015b20cd092c41efdef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:24.734265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.734976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-1538a86a-b498-4164-a55d-b9ff5e6d979d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:24.736627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1311/2653] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:24 2022] DELETE /identity/v3/projects/43cc64fc73814015b20cd092c41efdef => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:27:24.742142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9702a160-36b6-4860-a09f-3ffa98e8e8be None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:24.742645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9702a160-36b6-4860-a09f-3ffa98e8e8be None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:24.742753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9702a160-36b6-4860-a09f-3ffa98e8e8be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:24.761656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-9702a160-36b6-4860-a09f-3ffa98e8e8be None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:24.762896 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9702a160-36b6-4860-a09f-3ffa98e8e8be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:24.779560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:24.821936 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1343/2654] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:24.830705 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2c97d1-46d0-4c50-ac59-702046b7ca8a None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:24.831029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2c97d1-46d0-4c50-ac59-702046b7ca8a None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:24.831308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f2c97d1-46d0-4c50-ac59-702046b7ca8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:24.851334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-0f2c97d1-46d0-4c50-ac59-702046b7ca8a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:24.852534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0f2c97d1-46d0-4c50-ac59-702046b7ca8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:24.860390 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1312/2655] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:24 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:24.864661 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9702a160-36b6-4860-a09f-3ffa98e8e8be None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:24.875060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9702a160-36b6-4860-a09f-3ffa98e8e8be None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:24.879738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:24.880420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:24.880703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:24.880978 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:24.881730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:27:24.882039 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:24.883693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:24.895127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1344/2656] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:27:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:27:24.899824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2c97d1-46d0-4c50-ac59-702046b7ca8a None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:24.908891 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f2c97d1-46d0-4c50-ac59-702046b7ca8a None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:24.915021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:24.915609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:24.915887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:24.916159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:24.916842 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:24.919014 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:24.998102 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1313/2657] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:24 2022] POST /identity/v3/projects => generated 367 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:25.002336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.012322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af38aa1-6700-453f-b3c7-6d0e59c6fc3f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.069917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.070588 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.070906 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.071190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.071878 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.073797 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.167592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1345/2658] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:25 2022] POST /identity/v3/users => generated 284 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:25.171847 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.182111 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb47605c-6f6a-48ca-b782-758fc60efd20 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.186610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.187288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.187569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.187849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.188555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.190145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.200877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1314/2659] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:25.204903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.215426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f350e1-516d-4216-87de-b6f6cf5d0f68 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.220035 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19b3c4e1-725e-4292-8a5c-9efc8930a662 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.220689 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b3c4e1-725e-4292-8a5c-9efc8930a662 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.221004 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b3c4e1-725e-4292-8a5c-9efc8930a662 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.221281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19b3c4e1-725e-4292-8a5c-9efc8930a662 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.222003 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b3c4e1-725e-4292-8a5c-9efc8930a662 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.223642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19b3c4e1-725e-4292-8a5c-9efc8930a662 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.233482 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1346/2660] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:25.237375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.247362 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a568d4-440c-4e42-b373-a7e98128f236 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.251895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.252696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.252972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.253246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] PATH_INFO: `/v3/projects/6d9c8227f31b485980a1ead67bb860cf/users/84fcb3d8c5544bb7bf7ee8346bc1b322/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.268750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d9c8227f31b485980a1ead67bb860cf, user_id=84fcb3d8c5544bb7bf7ee8346bc1b322, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.271418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.374027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:DELETE=11,keystone:SELECT=85,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:25.378745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1315/2661] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:25 2022] PUT /identity/v3/projects/6d9c8227f31b485980a1ead67bb860cf/users/84fcb3d8c5544bb7bf7ee8346bc1b322/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:25.389373 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf7831d-27d2-49e0-a71e-49259d35a267 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.389670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf7831d-27d2-49e0-a71e-49259d35a267 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.389997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf7831d-27d2-49e0-a71e-49259d35a267 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.417891 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4bf7831d-27d2-49e0-a71e-49259d35a267 None None] MFA Rules not processed for user `84fcb3d8c5544bb7bf7ee8346bc1b322`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:25.419194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4bf7831d-27d2-49e0-a71e-49259d35a267 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:25.482944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1347/2662] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:25 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:25.487395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.549980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fa218d5-518a-457c-9259-5d80b94c79c9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.554649 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.555330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.555615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.555893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.556594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.558495 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.633047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1316/2663] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:25 2022] POST /identity/v3/projects => generated 369 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:25.637094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf7831d-27d2-49e0-a71e-49259d35a267 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.647736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf7831d-27d2-49e0-a71e-49259d35a267 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.660021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:25.704716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.705375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.705660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.705933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.706612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.708561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.792528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1348/2664] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:25 2022] POST /identity/v3/users => generated 291 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:27:25.796512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.808167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755a312a-8d98-4a66-8016-318dd5ec5bef admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.812798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.813458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.813744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.814023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.814755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.816389 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.829026 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1317/2665] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:25.833229 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:25.843636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d20093-d986-4993-b766-7d008ead8dce admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:25.848286 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:25.849368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:25.849653 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:25.850079 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] PATH_INFO: `/v3/projects/e41005e0b6a54e9d8d0ca70d8243fbda/users/ebe7112a695849a68135484c4afe54f3/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:25.865259 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e41005e0b6a54e9d8d0ca70d8243fbda, user_id=ebe7112a695849a68135484c4afe54f3, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:25.871500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:25.992647 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1349/2666] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:25 2022] PUT /identity/v3/projects/e41005e0b6a54e9d8d0ca70d8243fbda/users/ebe7112a695849a68135484c4afe54f3/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:25.997205 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:26.007188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e62ec1e-657d-4aaf-b108-66e1c26c8ad4 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:26.064016 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6f59546-b6fe-4fed-b6fd-6a92a74dd256 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:26.064642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f59546-b6fe-4fed-b6fd-6a92a74dd256 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:26.064926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f59546-b6fe-4fed-b6fd-6a92a74dd256 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:26.065203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f59546-b6fe-4fed-b6fd-6a92a74dd256 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:26.065928 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f59546-b6fe-4fed-b6fd-6a92a74dd256 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:26.067601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6f59546-b6fe-4fed-b6fd-6a92a74dd256 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:26.077537 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1318/2667] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:25 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:26.081601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:26.092265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa6a1203-0ea1-43a8-93a2-6d9579eaf8e6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:26.096910 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:26.097793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:26.098075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:26.098350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] PATH_INFO: `/v3/projects/e41005e0b6a54e9d8d0ca70d8243fbda/users/ebe7112a695849a68135484c4afe54f3/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:26.100970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e41005e0b6a54e9d8d0ca70d8243fbda, user_id=ebe7112a695849a68135484c4afe54f3, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:26.103731 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:26.113915 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1350/2668] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:26 2022] PUT /identity/v3/projects/e41005e0b6a54e9d8d0ca70d8243fbda/users/ebe7112a695849a68135484c4afe54f3/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:26.125180 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf026cfb-4b7f-4dd0-a6bc-9a55150137da None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:26.125533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf026cfb-4b7f-4dd0-a6bc-9a55150137da None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:26.125735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf026cfb-4b7f-4dd0-a6bc-9a55150137da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:26.152034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-bf026cfb-4b7f-4dd0-a6bc-9a55150137da None None] MFA Rules not processed for user `ebe7112a695849a68135484c4afe54f3`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:26.153409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-bf026cfb-4b7f-4dd0-a6bc-9a55150137da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:26.214232 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:26.223482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1319/2669] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:26 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:26.234725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:26.236556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:26.294921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2bd0a48-38b9-463e-903a-0311c58507d2 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:26.298610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6da2f26-a531-41fe-871b-a7a217fc2ee1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:26.299299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6da2f26-a531-41fe-871b-a7a217fc2ee1 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:26.299584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6da2f26-a531-41fe-871b-a7a217fc2ee1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:26.299862 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6da2f26-a531-41fe-871b-a7a217fc2ee1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:26.309006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6da2f26-a531-41fe-871b-a7a217fc2ee1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:26.310505 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6da2f26-a531-41fe-871b-a7a217fc2ee1 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:26.321230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1351/2670] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:26 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:29.135145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf026cfb-4b7f-4dd0-a6bc-9a55150137da None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:29.196983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf026cfb-4b7f-4dd0-a6bc-9a55150137da None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:29.201637 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:29.202503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:29.202809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:29.203093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] PATH_INFO: `/v3/users/84fcb3d8c5544bb7bf7ee8346bc1b322` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:29.204278 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=84fcb3d8c5544bb7bf7ee8346bc1b322)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:29.206309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:29.357292 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84fcb3d8c5544bb7bf7ee8346bc1b322'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:29.417971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84fcb3d8c5544bb7bf7ee8346bc1b322'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:29.418595 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84fcb3d8c5544bb7bf7ee8346bc1b322'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:29.425972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84fcb3d8c5544bb7bf7ee8346bc1b322'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:29.426679 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84fcb3d8c5544bb7bf7ee8346bc1b322'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:29.444743 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1320/2671] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:29 2022] DELETE /identity/v3/users/84fcb3d8c5544bb7bf7ee8346bc1b322 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:29.449303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:29.449793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:29.449902 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:29.467415 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:29.468610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:29.527163 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1352/2672] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:29.536741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:29.538677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:29.595559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c2d5c8-530a-4cdc-8647-16ac98973d03 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:29.599141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:29.599786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:29.600068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:29.600346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:29.609804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:29.611264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:29.621463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1321/2673] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:30.044773 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:30.046291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:30.058898 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73846fa-2c95-4d71-9399-574615905f2c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:30.063541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:30.066839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:30.070888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:30.070888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] PATH_INFO: `/v3/projects/6d9c8227f31b485980a1ead67bb860cf` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:30.070888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d9c8227f31b485980a1ead67bb860cf)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:30.071582 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:30.080055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:30.209309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d9c8227f31b485980a1ead67bb860cf', 'request_id': 'req-a569ee02-70e3-40e6-a600-3b845eeea8ec'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.289308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d9c8227f31b485980a1ead67bb860cf', 'request_id': 'req-a569ee02-70e3-40e6-a600-3b845eeea8ec'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.296041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d9c8227f31b485980a1ead67bb860cf', 'request_id': 'req-a569ee02-70e3-40e6-a600-3b845eeea8ec'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.307654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] The token cache is being invalidate because project 6d9c8227f31b485980a1ead67bb860cf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:30.307942 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.308563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.310092 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1353/2674] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:30 2022] DELETE /identity/v3/projects/6d9c8227f31b485980a1ead67bb860cf => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:30.314040 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:30.323888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f68028d-16f1-45bc-9f48-579b3dda8870 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:30.336613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:30.381241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:30.382291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:30.382578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:30.382892 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] PATH_INFO: `/v3/users/ebe7112a695849a68135484c4afe54f3` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:30.384089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ebe7112a695849a68135484c4afe54f3)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:30.386053 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:30.622654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ebe7112a695849a68135484c4afe54f3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.682838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ebe7112a695849a68135484c4afe54f3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.683452 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ebe7112a695849a68135484c4afe54f3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.690866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ebe7112a695849a68135484c4afe54f3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.691501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-8c8a9c6f-b91a-434a-8136-7aa651768351 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ebe7112a695849a68135484c4afe54f3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:30.709022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1322/2675] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:30 2022] DELETE /identity/v3/users/ebe7112a695849a68135484c4afe54f3 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:31.141557 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:31.143467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:31.166449 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a569ee02-70e3-40e6-a600-3b845eeea8ec admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:31.231866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:31.232730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.233015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.233292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] PATH_INFO: `/v3/projects/e41005e0b6a54e9d8d0ca70d8243fbda` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:31.234471 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e41005e0b6a54e9d8d0ca70d8243fbda)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:31.236470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:31.355611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e41005e0b6a54e9d8d0ca70d8243fbda', 'request_id': 'req-f94ae802-7bb3-4460-85b1-6e3f98d1d248'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:31.440523 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e41005e0b6a54e9d8d0ca70d8243fbda', 'request_id': 'req-f94ae802-7bb3-4460-85b1-6e3f98d1d248'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:31.449096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e41005e0b6a54e9d8d0ca70d8243fbda', 'request_id': 'req-f94ae802-7bb3-4460-85b1-6e3f98d1d248'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:31.459787 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] The token cache is being invalidate because project e41005e0b6a54e9d8d0ca70d8243fbda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:31.460184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:31.460847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f94ae802-7bb3-4460-85b1-6e3f98d1d248 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:31.462467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1354/2676] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:31 2022] DELETE /identity/v3/projects/e41005e0b6a54e9d8d0ca70d8243fbda => generated 0 bytes in 321 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:31.473169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee503192-ccf7-4472-a199-e64e63cdbf35 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.473169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee503192-ccf7-4472-a199-e64e63cdbf35 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.473169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee503192-ccf7-4472-a199-e64e63cdbf35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:31.486343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ee503192-ccf7-4472-a199-e64e63cdbf35 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:31.487824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ee503192-ccf7-4472-a199-e64e63cdbf35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:31.554559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1323/2677] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:31.562150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1f67b-fa4d-4a6e-80d6-99647606bcc2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.562430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1f67b-fa4d-4a6e-80d6-99647606bcc2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.562702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c1f67b-fa4d-4a6e-80d6-99647606bcc2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:31.585276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d7c1f67b-fa4d-4a6e-80d6-99647606bcc2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:31.585276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d7c1f67b-fa4d-4a6e-80d6-99647606bcc2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:31.591073 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1355/2678] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:31.595093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee503192-ccf7-4472-a199-e64e63cdbf35 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:31.605414 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee503192-ccf7-4472-a199-e64e63cdbf35 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:31.609971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:31.610627 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.610937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.611213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:31.611934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:27:31.612239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:31.613836 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:31.626795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1324/2679] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:27:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:27:31.630912 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c1f67b-fa4d-4a6e-80d6-99647606bcc2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:31.640285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c1f67b-fa4d-4a6e-80d6-99647606bcc2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:31.646170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:31.646972 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.647917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.648294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:31.648987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:31.654460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:31.757180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:31.757180 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1356/2680] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:31 2022] POST /identity/v3/projects => generated 367 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:27:31.757610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:31.772031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d8dd86-3e3f-4913-ba10-b7d451c41063 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:31.794826 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:31.831676 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:31.832355 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.832633 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.832910 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:31.833612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:31.836577 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:31.935012 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1325/2681] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:31 2022] POST /identity/v3/users => generated 284 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:31.941439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:31.951920 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02eef4c-1754-423a-b4fd-bb1f94b7e22b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:31.962687 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:31.963395 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.963682 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.963960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:31.964735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:31.966341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:31.980623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1357/2682] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:31 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:31.981756 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:31.992200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82956fcf-ffb5-4602-a42f-38bae3ee6214 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:31.998607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b10498-d5d4-4ca8-a00f-2234125e470b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:31.999312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b10498-d5d4-4ca8-a00f-2234125e470b admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:31.999645 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b10498-d5d4-4ca8-a00f-2234125e470b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:31.999927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b10498-d5d4-4ca8-a00f-2234125e470b admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.000662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b10498-d5d4-4ca8-a00f-2234125e470b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.002263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b10498-d5d4-4ca8-a00f-2234125e470b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.016591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1326/2683] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:31 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:27:32.031484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:32.031484 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f78fbec-a896-4499-8eb5-911655a993d3 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:32.034673 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:32.035636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.035914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.036194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] PATH_INFO: `/v3/projects/2ec112df9b1140609d0043b8fcb3f771/users/1835ce14fd9c40eab62da7deee2f1c47/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.052888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ec112df9b1140609d0043b8fcb3f771, user_id=1835ce14fd9c40eab62da7deee2f1c47, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.055796 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.117722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1358/2684] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:32 2022] PUT /identity/v3/projects/2ec112df9b1140609d0043b8fcb3f771/users/1835ce14fd9c40eab62da7deee2f1c47/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:32.133828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ad721d-1095-4117-b83f-769b0f3a6687 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.134235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ad721d-1095-4117-b83f-769b0f3a6687 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.134591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70ad721d-1095-4117-b83f-769b0f3a6687 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.166052 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-70ad721d-1095-4117-b83f-769b0f3a6687 None None] MFA Rules not processed for user `1835ce14fd9c40eab62da7deee2f1c47`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:32.167562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-70ad721d-1095-4117-b83f-769b0f3a6687 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:32.231802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1327/2685] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:32 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:32.236214 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:32.300592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d369c2e6-d30b-4c37-a69c-42e2aac50cee admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:32.305230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:32.305884 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.306167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.306499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.307269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.309220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.380623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1359/2686] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:32 2022] POST /identity/v3/projects => generated 365 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:27:32.385340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ad721d-1095-4117-b83f-769b0f3a6687 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:32.396127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70ad721d-1095-4117-b83f-769b0f3a6687 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:32.453530 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:32.454204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.454492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.454800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.455546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.457509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.546203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1328/2687] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:32 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:32.550120 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:32.561291 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe0398e-f78e-439f-9c1c-cedbb4bbca5e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:32.565879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:32.566531 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.570143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.570143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.570143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.570143 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.580486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1360/2688] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:32.584551 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:32.594454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b1fe84-d229-4afb-bc93-7be7e02b20e7 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:32.600426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:32.600426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.600426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.600750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] PATH_INFO: `/v3/projects/ae5fdbc61ed046219d076b89cbaf049d/users/fefc6000d95f4c79856f3a2334cc4fb8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.614913 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae5fdbc61ed046219d076b89cbaf049d, user_id=fefc6000d95f4c79856f3a2334cc4fb8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.618409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.712505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:32.718391 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1329/2689] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:32 2022] PUT /identity/v3/projects/ae5fdbc61ed046219d076b89cbaf049d/users/fefc6000d95f4c79856f3a2334cc4fb8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:32.725766 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:32.733527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d690808-a1db-44c5-8e25-2b26a8d42717 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:32.789888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785fbc63-a0ac-46fb-8a36-995365aacd2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:32.790552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fbc63-a0ac-46fb-8a36-995365aacd2e admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.790941 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fbc63-a0ac-46fb-8a36-995365aacd2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.791222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785fbc63-a0ac-46fb-8a36-995365aacd2e admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.791953 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785fbc63-a0ac-46fb-8a36-995365aacd2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.793869 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-785fbc63-a0ac-46fb-8a36-995365aacd2e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.804642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1361/2690] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:32 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:32.808832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:32.819379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a31076b-bf8e-4e65-b8c5-c82de0f92175 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:32.824294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:32.825587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.825870 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.826166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] PATH_INFO: `/v3/projects/ae5fdbc61ed046219d076b89cbaf049d/users/fefc6000d95f4c79856f3a2334cc4fb8/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.828251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae5fdbc61ed046219d076b89cbaf049d, user_id=fefc6000d95f4c79856f3a2334cc4fb8, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:32.831002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:32.841454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1330/2691] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:32 2022] PUT /identity/v3/projects/ae5fdbc61ed046219d076b89cbaf049d/users/fefc6000d95f4c79856f3a2334cc4fb8/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:32.906221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f118c17b-b6d3-4e2e-8386-4016639ed077 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:32.906507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f118c17b-b6d3-4e2e-8386-4016639ed077 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:32.906820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f118c17b-b6d3-4e2e-8386-4016639ed077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:32.934111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f118c17b-b6d3-4e2e-8386-4016639ed077 None None] MFA Rules not processed for user `fefc6000d95f4c79856f3a2334cc4fb8`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:32.935342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f118c17b-b6d3-4e2e-8386-4016639ed077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:32.954546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:33.002047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1362/2692] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:32 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:33.013241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:33.014952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:33.070962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac150c2d-cfa2-4ca8-a905-cb3a7aa724a2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:33.074460 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:33.075122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:33.075407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:33.075684 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:33.084563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:33.086896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:33.096150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1331/2693] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:33 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:33.287323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f118c17b-b6d3-4e2e-8386-4016639ed077 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:33.298418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f118c17b-b6d3-4e2e-8386-4016639ed077 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:33.302048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa645cf-d899-4c7b-830f-a08f2f6f48b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:33.302695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa645cf-d899-4c7b-830f-a08f2f6f48b8 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:33.303232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa645cf-d899-4c7b-830f-a08f2f6f48b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:33.303720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa645cf-d899-4c7b-830f-a08f2f6f48b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:33.363469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa645cf-d899-4c7b-830f-a08f2f6f48b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:33.364916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aa645cf-d899-4c7b-830f-a08f2f6f48b8 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:33.374353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1363/2694] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:33 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:36.927801 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:36.929421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:36.994986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ecb8f6-5df7-4553-ba65-6f29b918be8f service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:36.999849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:37.000731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:37.001017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:37.001294 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] PATH_INFO: `/v3/users/0d1fdf3c170249eebf03b8a29b4f47e3` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:37.002510 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d1fdf3c170249eebf03b8a29b4f47e3)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:37.004487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:37.239340 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d1fdf3c170249eebf03b8a29b4f47e3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:37.316760 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d1fdf3c170249eebf03b8a29b4f47e3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:37.317387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d1fdf3c170249eebf03b8a29b4f47e3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:37.325474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d1fdf3c170249eebf03b8a29b4f47e3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:37.326114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d1fdf3c170249eebf03b8a29b4f47e3'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:37.344049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1332/2695] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:36 2022] DELETE /identity/v3/users/0d1fdf3c170249eebf03b8a29b4f47e3 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:37.348606 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:37.349097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:37.349208 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:37.366199 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:37.367961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:37.427496 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1364/2696] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:27:37.437831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:37.439272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:37.449594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:37.507781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1a7ba2-13b8-449a-9c79-65f41c76752c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:37.511871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:37.512548 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:37.512830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:37.513104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:37.522619 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:37.525043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:37.534727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1333/2697] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:37.951554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:37.953027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:37.966246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a1fd8e7-e0dd-412d-b47e-f57fea9b39fb None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:37.971015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:37.971886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:37.972173 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:37.972453 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] PATH_INFO: `/v3/projects/b5c11acfe02f4346aa5d6ecc25a5d07c` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:37.973650 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b5c11acfe02f4346aa5d6ecc25a5d07c)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:37.975630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:38.104916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b5c11acfe02f4346aa5d6ecc25a5d07c', 'request_id': 'req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:38.107508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:DELETE=4,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:38.225364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b5c11acfe02f4346aa5d6ecc25a5d07c', 'request_id': 'req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:38.235934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b5c11acfe02f4346aa5d6ecc25a5d07c', 'request_id': 'req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:38.250991 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] The token cache is being invalidate because project b5c11acfe02f4346aa5d6ecc25a5d07c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:38.251289 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:38.252760 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3afe83c2-52a2-4f7f-9d20-ed67da8151e1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:38.256310 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1365/2698] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:37 2022] DELETE /identity/v3/projects/b5c11acfe02f4346aa5d6ecc25a5d07c => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:40.001153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:40.002614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:40.015384 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b6934-c8c1-437a-ae74-85f638aa506c service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:40.070621 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:40.071668 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:40.072084 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:40.072489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] PATH_INFO: `/v3/users/1835ce14fd9c40eab62da7deee2f1c47` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:40.074080 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1835ce14fd9c40eab62da7deee2f1c47)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:40.076094 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:40.265347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1835ce14fd9c40eab62da7deee2f1c47'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:40.345083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1835ce14fd9c40eab62da7deee2f1c47'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:40.345710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1835ce14fd9c40eab62da7deee2f1c47'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:40.353909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1835ce14fd9c40eab62da7deee2f1c47'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:40.354549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1835ce14fd9c40eab62da7deee2f1c47'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:40.371962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1334/2699] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:40 2022] DELETE /identity/v3/users/1835ce14fd9c40eab62da7deee2f1c47 => generated 0 bytes in 371 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:40.376660 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:40.377170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:40.377170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:40.394937 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:40.396141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:40.454161 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1366/2700] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:40.464227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:40.465631 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:40.497980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:40.522958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a522a2-980f-4ec4-9188-d380f5075868 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:40.527319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:40.528226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:40.528865 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:40.529707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:40.539188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:40.541462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:40.551593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1335/2701] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:41.187572 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:41.189098 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:41.201847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfec7889-9344-451d-af9d-7ec9ee412217 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:41.206850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:41.207876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:41.208162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:41.208499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] PATH_INFO: `/v3/projects/2ec112df9b1140609d0043b8fcb3f771` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:41.210407 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ec112df9b1140609d0043b8fcb3f771)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:41.212526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:41.351750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ec112df9b1140609d0043b8fcb3f771', 'request_id': 'req-3bdbc510-ede9-4d67-8d3f-893403d09485'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.442720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ec112df9b1140609d0043b8fcb3f771', 'request_id': 'req-3bdbc510-ede9-4d67-8d3f-893403d09485'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.450185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ec112df9b1140609d0043b8fcb3f771', 'request_id': 'req-3bdbc510-ede9-4d67-8d3f-893403d09485'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.460867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] The token cache is being invalidate because project 2ec112df9b1140609d0043b8fcb3f771 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:41.461097 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.461761 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.464037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1367/2702] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:41 2022] DELETE /identity/v3/projects/2ec112df9b1140609d0043b8fcb3f771 => generated 0 bytes in 277 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:41.468167 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:41.478188 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b177ae8e-bd14-4a4b-ac5d-aacb2d888a78 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:41.538536 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:41.539450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:41.539938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:41.540379 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] PATH_INFO: `/v3/users/fefc6000d95f4c79856f3a2334cc4fb8` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:41.541830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fefc6000d95f4c79856f3a2334cc4fb8)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:41.544073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:41.778954 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fefc6000d95f4c79856f3a2334cc4fb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.831514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fefc6000d95f4c79856f3a2334cc4fb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.832625 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fefc6000d95f4c79856f3a2334cc4fb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.843851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fefc6000d95f4c79856f3a2334cc4fb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.844894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f55c00f-d9e4-4ee4-ad91-246fb16ffc70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fefc6000d95f4c79856f3a2334cc4fb8'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:41.873858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1336/2703] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:41 2022] DELETE /identity/v3/users/fefc6000d95f4c79856f3a2334cc4fb8 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:27:42.302425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:42.304041 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:42.328447 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdbc510-ede9-4d67-8d3f-893403d09485 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:42.403349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:42.404249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:42.404527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:42.405921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] PATH_INFO: `/v3/projects/ae5fdbc61ed046219d076b89cbaf049d` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:42.406066 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae5fdbc61ed046219d076b89cbaf049d)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:42.407975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:42.414504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:42.559629 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae5fdbc61ed046219d076b89cbaf049d', 'request_id': 'req-6ba4d0c1-4b78-4942-9674-49e01d930939'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:42.636578 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae5fdbc61ed046219d076b89cbaf049d', 'request_id': 'req-6ba4d0c1-4b78-4942-9674-49e01d930939'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:42.644428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae5fdbc61ed046219d076b89cbaf049d', 'request_id': 'req-6ba4d0c1-4b78-4942-9674-49e01d930939'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:42.657462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] The token cache is being invalidate because project ae5fdbc61ed046219d076b89cbaf049d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:42.657794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:42.658478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ba4d0c1-4b78-4942-9674-49e01d930939 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:42.660251 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1368/2704] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:42 2022] DELETE /identity/v3/projects/ae5fdbc61ed046219d076b89cbaf049d => generated 0 bytes in 359 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:42.665785 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c958f196-fad1-4d24-9867-a7e20db1bc9f None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:42.666266 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c958f196-fad1-4d24-9867-a7e20db1bc9f None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:42.666571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c958f196-fad1-4d24-9867-a7e20db1bc9f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:42.691249 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c958f196-fad1-4d24-9867-a7e20db1bc9f None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:42.692525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c958f196-fad1-4d24-9867-a7e20db1bc9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:42.750065 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:42.764493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1337/2705] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:42.772233 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d5c74-d125-4adf-bc7d-913c3fee6c1e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:42.772601 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d5c74-d125-4adf-bc7d-913c3fee6c1e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:42.772955 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298d5c74-d125-4adf-bc7d-913c3fee6c1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:42.793265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-298d5c74-d125-4adf-bc7d-913c3fee6c1e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:42.801488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-298d5c74-d125-4adf-bc7d-913c3fee6c1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:42.802519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1369/2706] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:42.807338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c958f196-fad1-4d24-9867-a7e20db1bc9f None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:42.817254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c958f196-fad1-4d24-9867-a7e20db1bc9f None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:42.822524 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:42.823212 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:42.823545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:42.823813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:42.824553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:27:42.824860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:42.826453 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:42.839472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1338/2707] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:27:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:27:42.843983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d5c74-d125-4adf-bc7d-913c3fee6c1e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:42.853576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298d5c74-d125-4adf-bc7d-913c3fee6c1e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:42.860228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:42.861321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:42.863269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:42.863269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:42.863269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:42.864535 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:43.003939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1370/2708] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:27:42 2022] POST /identity/v3/projects => generated 375 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:43.008490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:43.019514 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfe4a1a-ec36-4731-8a4a-10d84fb0cab8 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:43.078955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:43.079556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:43.079889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:43.080285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:43.081006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:43.082958 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:43.311927 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1339/2709] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:27:43 2022] POST /identity/v3/users => generated 288 bytes in 304 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:27:43.317932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:43.328751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec024bd8-3bba-432b-82d5-b36ae83a20d8 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:43.333370 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:43.333970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:43.334276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:43.334560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:43.335333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:43.337135 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:43.347961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1371/2710] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:43 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:43.351972 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:43.362039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d06011-1451-413e-b315-8b34b3c5ab9c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:43.366619 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134bb2b3-397e-4cca-acba-eccfe08a1368 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:43.367315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134bb2b3-397e-4cca-acba-eccfe08a1368 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:43.367602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134bb2b3-397e-4cca-acba-eccfe08a1368 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:43.370013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134bb2b3-397e-4cca-acba-eccfe08a1368 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:43.370013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134bb2b3-397e-4cca-acba-eccfe08a1368 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:43.370732 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134bb2b3-397e-4cca-acba-eccfe08a1368 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:43.381317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1340/2711] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:27:43 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:27:43.386190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:43.396190 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a2f0577-592d-4daf-a45f-5bf3af876a2f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:43.401060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:43.401959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:43.402320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:43.402602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] PATH_INFO: `/v3/projects/a0bec7f0bfe84b8aa3a341fd23de08b2/users/3c2fb1f6dcba47889c00275eacb525fe/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:43.415408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2bb93a-4acf-44f7-8ad0-053fd8a6fb61 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:43.415685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2bb93a-4acf-44f7-8ad0-053fd8a6fb61 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:43.415991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2bb93a-4acf-44f7-8ad0-053fd8a6fb61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:43.421399 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a0bec7f0bfe84b8aa3a341fd23de08b2, user_id=3c2fb1f6dcba47889c00275eacb525fe, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:43.424136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:43.434874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-be2bb93a-4acf-44f7-8ad0-053fd8a6fb61 None None] MFA Rules not processed for user `d0adaa092d944202b3279d10e0f94a91`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:43.441182 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-be2bb93a-4acf-44f7-8ad0-053fd8a6fb61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:43.490798 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1341/2712] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:43 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:27:43.517281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1372/2713] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:27:43 2022] PUT /identity/v3/projects/a0bec7f0bfe84b8aa3a341fd23de08b2/users/3c2fb1f6dcba47889c00275eacb525fe/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:43.529581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414c9c3-8821-477b-9a69-3f1c8d8a1793 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:43.529581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414c9c3-8821-477b-9a69-3f1c8d8a1793 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:43.529581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c414c9c3-8821-477b-9a69-3f1c8d8a1793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:43.556855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-c414c9c3-8821-477b-9a69-3f1c8d8a1793 None None] MFA Rules not processed for user `3c2fb1f6dcba47889c00275eacb525fe`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:43.558178 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c414c9c3-8821-477b-9a69-3f1c8d8a1793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:43.623095 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:43.640189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1342/2714] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:43 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:27:43.651052 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:43.652987 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:43.720515 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85ae5255-7abf-490f-9bac-77372d18cb06 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:43.724451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40aea45c-5eea-4374-8cfd-cb93de9b71bd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:43.725125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aea45c-5eea-4374-8cfd-cb93de9b71bd service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:43.725412 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aea45c-5eea-4374-8cfd-cb93de9b71bd service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:43.725693 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40aea45c-5eea-4374-8cfd-cb93de9b71bd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:43.741975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aea45c-5eea-4374-8cfd-cb93de9b71bd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:43.744145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40aea45c-5eea-4374-8cfd-cb93de9b71bd service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:43.763658 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1373/2715] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:43 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:47.858075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414c9c3-8821-477b-9a69-3f1c8d8a1793 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:47.973542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c414c9c3-8821-477b-9a69-3f1c8d8a1793 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:47.982296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:47.987105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:47.987105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:47.987105 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] PATH_INFO: `/v3/users/3c2fb1f6dcba47889c00275eacb525fe` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:47.987895 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3c2fb1f6dcba47889c00275eacb525fe)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:47.990199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:48.236616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3c2fb1f6dcba47889c00275eacb525fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:48.362368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3c2fb1f6dcba47889c00275eacb525fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:48.363089 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3c2fb1f6dcba47889c00275eacb525fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:48.371535 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3c2fb1f6dcba47889c00275eacb525fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:48.372258 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3c2fb1f6dcba47889c00275eacb525fe'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:48.404945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1343/2716] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:27:47 2022] DELETE /identity/v3/users/3c2fb1f6dcba47889c00275eacb525fe => generated 0 bytes in 547 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:48.409561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:48.410072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:48.410315 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:48.429262 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:27:48.432919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:48.467559 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:48.506697 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1374/2717] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:27:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:27:48.517156 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:48.518866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:48.596872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-267ebb9c-6a63-4054-80d9-5bfeadce5185 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:48.601017 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23f85aa9-beef-4f80-b40a-110a0b00a905 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:48.601688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f85aa9-beef-4f80-b40a-110a0b00a905 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:48.602077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f85aa9-beef-4f80-b40a-110a0b00a905 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:48.603352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23f85aa9-beef-4f80-b40a-110a0b00a905 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:48.615022 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f85aa9-beef-4f80-b40a-110a0b00a905 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:48.616513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23f85aa9-beef-4f80-b40a-110a0b00a905 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:48.629096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1344/2718] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:27:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:27:49.322475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:27:49.325909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:27:49.342919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a73bff-4baf-4b43-8c90-f5ebb89bf4c9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:27:49.347801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:27:49.348757 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:27:49.348921 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:27:49.349789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] PATH_INFO: `/v3/projects/a0bec7f0bfe84b8aa3a341fd23de08b2` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:27:49.351341 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a0bec7f0bfe84b8aa3a341fd23de08b2)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:27:49.355312 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:27:49.491103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a0bec7f0bfe84b8aa3a341fd23de08b2', 'request_id': 'req-827aa4a5-cde3-4d92-b974-bfca8b345107'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:49.592282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a0bec7f0bfe84b8aa3a341fd23de08b2', 'request_id': 'req-827aa4a5-cde3-4d92-b974-bfca8b345107'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:49.601317 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a0bec7f0bfe84b8aa3a341fd23de08b2', 'request_id': 'req-827aa4a5-cde3-4d92-b974-bfca8b345107'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:49.620609 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] The token cache is being invalidate because project a0bec7f0bfe84b8aa3a341fd23de08b2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:27:49.620919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:49.621723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:27:49.627755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1375/2719] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:27:49 2022] DELETE /identity/v3/projects/a0bec7f0bfe84b8aa3a341fd23de08b2 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:27:58.622512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=83,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:27:59.616349 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=39,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:28:01.267254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51f85de-21af-476c-9917-92dd579a29af None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:28:01.267786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51f85de-21af-476c-9917-92dd579a29af None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:28:01.267890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51f85de-21af-476c-9917-92dd579a29af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:28:01.285387 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-a51f85de-21af-476c-9917-92dd579a29af None None] MFA Rules not processed for user `8d824401d1454a2bb19d7628e38b256a`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:28:01.286691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a51f85de-21af-476c-9917-92dd579a29af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:28:01.342659 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1345/2720] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:28:01 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:28:11.336401 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=25 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:28:59.279767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:28:59.281209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:28:59.340236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827aa4a5-cde3-4d92-b974-bfca8b345107 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:28:59.343793 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:28:59.344435 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:28:59.344722 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:28:59.345001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:28:59.346748 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:28:59.428517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:28:59.430111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:28:59.450537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1376/2721] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:28:59 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:29:08.809363 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51f85de-21af-476c-9917-92dd579a29af None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:29:08.819730 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51f85de-21af-476c-9917-92dd579a29af None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:29:08.823296 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-875313a6-b58c-4ac3-a2c8-34416dba2a89 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:29:08.823933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875313a6-b58c-4ac3-a2c8-34416dba2a89 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:08.824216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875313a6-b58c-4ac3-a2c8-34416dba2a89 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:08.824498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-875313a6-b58c-4ac3-a2c8-34416dba2a89 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:08.827036 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-875313a6-b58c-4ac3-a2c8-34416dba2a89 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:08.840245 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875313a6-b58c-4ac3-a2c8-34416dba2a89 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:29:08.841735 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-875313a6-b58c-4ac3-a2c8-34416dba2a89 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:29:08.852572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1346/2722] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:29:08 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:29:09.447944 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=63 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:29:14.699326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:29:14.702093 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:14.716292 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4949f9b7-c4db-4af7-b33e-3f3ca95d3032 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:29:14.726136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:29:14.726136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:14.726136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:14.726136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:14.726849 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:14.748198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:29:14.748198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:29:14.759462 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1377/2723] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:29:14 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 61 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:29:18.847766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=6 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:29:19.738995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:19.754006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:19.754006 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:19.757087 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] MFA Rules not processed for user `8d824401d1454a2bb19d7628e38b256a`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:29:19.758304 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:19.765280 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1347/2724] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:29:19 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:29:20.068926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:29:20.070404 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:20.125860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dbcc184-6abd-401c-9b61-df4aee5cc660 service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:29:20.129490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:29:20.130156 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:20.130441 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:20.130716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:20.132511 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:20.142575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:29:20.144047 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:29:20.152155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1378/2725] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:29:20 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:29:20.551079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:29:20.555874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:20.627749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618e283a-4ae8-48a3-8732-be90395ccd49 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:29:20.632672 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:29:20.633325 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:20.633614 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:20.633880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:20.635707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:20.646902 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:29:20.648481 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:29:20.658790 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1348/2726] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:29:20 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:29:23.584816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:29:23.594311 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879f8fdf-9c8f-4426-b924-098afb5402a9 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:29:23.597950 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:29:23.598583 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:23.598894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:23.599167 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:23.600940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:23.611534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:29:23.614755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:29:23.621144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1379/2727] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:29:23 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:29:30.654000 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=35 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:29:31.457504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:29:31.459046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:31.474707 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ed6276-cefb-4d06-8629-e1bde8f1ee49 service placement] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:29:31.480219 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29f04508-5a70-4d44-a0ef-21c4268aca6a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:29:31.480862 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f04508-5a70-4d44-a0ef-21c4268aca6a service nova] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:31.481146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f04508-5a70-4d44-a0ef-21c4268aca6a service nova] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:31.481438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29f04508-5a70-4d44-a0ef-21c4268aca6a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:31.483409 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-29f04508-5a70-4d44-a0ef-21c4268aca6a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:31.495752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f04508-5a70-4d44-a0ef-21c4268aca6a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:29:31.497334 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29f04508-5a70-4d44-a0ef-21c4268aca6a service nova] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:29:31.508177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1349/2728] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:29:31 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:29:33.617402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=38 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:29:40.984617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:29:40.984617 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:40.989714 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3c9da7-e618-4bab-9060-b5fddaf95082 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:29:40.993410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:29:40.994084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:40.994366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:40.994646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:40.996422 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:41.056298 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:29:41.057735 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:29:41.067602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1380/2729] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:29:40 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:29:41.503610 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=6 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:29:49.093567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c8baa-6903-440a-a749-50971ff7f0a9 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:29:49.094227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c8baa-6903-440a-a749-50971ff7f0a9 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:29:49.094227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e32c8baa-6903-440a-a749-50971ff7f0a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:29:49.111680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e32c8baa-6903-440a-a749-50971ff7f0a9 None None] MFA Rules not processed for user `d0adaa092d944202b3279d10e0f94a91`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:29:49.112893 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e32c8baa-6903-440a-a749-50971ff7f0a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:29:49.119753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1350/2730] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:29:49 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 27 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:29:51.063247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=28 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:29:59.106796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:18.983345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:18.990220 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3555f0c6-a4ce-46fe-955a-6a896162569e service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:18.994367 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:18.995084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:18.995396 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:18.995745 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:18.997580 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:19.008688 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:19.010194 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:19.019307 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1381/2731] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:18 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:30:29.015248 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=6 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:41.914597 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32c8baa-6903-440a-a749-50971ff7f0a9 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:41.924957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e32c8baa-6903-440a-a749-50971ff7f0a9 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:41.928436 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:41.929097 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:41.929375 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:41.929646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:41.931383 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:41.993505 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:41.994877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:42.005035 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1351/2732] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:30:45.679581 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:45.681566 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:45.700742 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54da2840-3eeb-48f8-8760-d81346f74531 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:45.709200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:45.709200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:45.709200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:45.709200 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:45.710192 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:45.733466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:45.735248 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:45.746011 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1382/2733] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:45 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:30:45.831646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:45.837300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:45.849118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d079ddfe-12a1-451b-ac04-53688c2f3977 service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:45.854288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:45.855223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:45.855534 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:45.855991 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] PATH_INFO: `/v3/users/8d824401d1454a2bb19d7628e38b256a` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:45.857404 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8d824401d1454a2bb19d7628e38b256a)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:45.859473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:46.086207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8d824401d1454a2bb19d7628e38b256a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.170174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8d824401d1454a2bb19d7628e38b256a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.170834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8d824401d1454a2bb19d7628e38b256a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.180066 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8d824401d1454a2bb19d7628e38b256a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.180828 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-aedfc179-3985-411d-b5be-32fec4ed24c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8d824401d1454a2bb19d7628e38b256a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.205689 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1352/2734] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:30:45 2022] DELETE /identity/v3/users/8d824401d1454a2bb19d7628e38b256a => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:46.509174 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:46.526932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a5123a-4eac-4d65-8238-25d52bd02de6 service placement] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:46.597663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:46.597663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:46.597663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:46.597663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] PATH_INFO: `/v3/projects/cc4efb6abcf3412897b7f1484988aa2b` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:46.598155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc4efb6abcf3412897b7f1484988aa2b)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:46.598155 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:46.754498 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc4efb6abcf3412897b7f1484988aa2b', 'request_id': 'req-e17d8b43-18d9-4b48-8a71-56070974267c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.841670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc4efb6abcf3412897b7f1484988aa2b', 'request_id': 'req-e17d8b43-18d9-4b48-8a71-56070974267c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.849043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc4efb6abcf3412897b7f1484988aa2b', 'request_id': 'req-e17d8b43-18d9-4b48-8a71-56070974267c'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.860139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] The token cache is being invalidate because project cc4efb6abcf3412897b7f1484988aa2b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:30:46.860428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.861091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-e17d8b43-18d9-4b48-8a71-56070974267c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:46.866228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1383/2735] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:30:46 2022] DELETE /identity/v3/projects/cc4efb6abcf3412897b7f1484988aa2b => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:46.867758 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9c988-06a1-4042-915e-2faa8a773c82 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:46.868254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9c988-06a1-4042-915e-2faa8a773c82 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:46.868368 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90f9c988-06a1-4042-915e-2faa8a773c82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:46.888550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-90f9c988-06a1-4042-915e-2faa8a773c82 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:46.889774 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-90f9c988-06a1-4042-915e-2faa8a773c82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:46.954100 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1353/2736] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:46.962328 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f170e65-cf0e-4d7c-89ff-a85f0d0d68aa None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:46.962616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f170e65-cf0e-4d7c-89ff-a85f0d0d68aa None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:46.962940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f170e65-cf0e-4d7c-89ff-a85f0d0d68aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:46.982026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-6f170e65-cf0e-4d7c-89ff-a85f0d0d68aa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:46.983321 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6f170e65-cf0e-4d7c-89ff-a85f0d0d68aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:46.991817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1384/2737] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:46.996733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f9c988-06a1-4042-915e-2faa8a773c82 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.006872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90f9c988-06a1-4042-915e-2faa8a773c82 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.011620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.012886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.013166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.013443 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.014171 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:30:47.014484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.016127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.017946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:47.028795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1354/2738] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:30:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:30:47.036166 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f170e65-cf0e-4d7c-89ff-a85f0d0d68aa None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.047285 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f170e65-cf0e-4d7c-89ff-a85f0d0d68aa None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.052121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.052786 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.053067 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.053340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.054024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.055956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.154670 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1385/2739] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:30:47 2022] POST /identity/v3/projects => generated 387 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:30:47.158985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.171176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeda6672-d91c-457c-aacc-666f5fc06866 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.231556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.232324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.232613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.233031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.233744 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.235720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.316800 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1355/2740] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:30:47 2022] POST /identity/v3/users => generated 294 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:30:47.321036 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.331990 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc550d09-165c-49da-8544-9e63591d1182 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.337318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.338108 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.338418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.338698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.339918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.341905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.352727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1386/2741] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:30:47.357750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.368270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae9cf946-e7f4-480d-9de1-83b5deab147e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.373932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4467616-1323-4619-a5dc-73172c30ec07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.374691 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4467616-1323-4619-a5dc-73172c30ec07 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.375003 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4467616-1323-4619-a5dc-73172c30ec07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.375282 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4467616-1323-4619-a5dc-73172c30ec07 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.376034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4467616-1323-4619-a5dc-73172c30ec07 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.377733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4467616-1323-4619-a5dc-73172c30ec07 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.392148 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1356/2742] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:30:47.393293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.405743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3bc919-4452-4a53-b2c2-6ee627af7cae admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.410525 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.411623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.412082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.412425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] PATH_INFO: `/v3/projects/84a6ecf0fc1b48ebb7f45e4b657859fc/users/c8387cc5880c491a82875e53ecd2aa43/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.430861 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] RBAC: Authorizing `identity:create_grant(project_id=84a6ecf0fc1b48ebb7f45e4b657859fc, user_id=c8387cc5880c491a82875e53ecd2aa43, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.433607 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.542213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1387/2743] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:30:47 2022] PUT /identity/v3/projects/84a6ecf0fc1b48ebb7f45e4b657859fc/users/c8387cc5880c491a82875e53ecd2aa43/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:30:47.554201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ea295-d61e-4cf7-bd3a-cf4e2ce3031d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.554513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ea295-d61e-4cf7-bd3a-cf4e2ce3031d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.555039 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-081ea295-d61e-4cf7-bd3a-cf4e2ce3031d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.586507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-081ea295-d61e-4cf7-bd3a-cf4e2ce3031d None None] MFA Rules not processed for user `c8387cc5880c491a82875e53ecd2aa43`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:47.587484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-081ea295-d61e-4cf7-bd3a-cf4e2ce3031d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:47.654692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1357/2744] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:47.659048 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.664838 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:47.726128 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217b05c3-b9b5-49ed-9ba3-b9254f67e89c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.731965 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.732751 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.733101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.733478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.734408 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.736470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.835144 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1388/2745] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:30:47 2022] POST /identity/v3/projects => generated 385 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:30:47.839223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ea295-d61e-4cf7-bd3a-cf4e2ce3031d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:47.851418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-081ea295-d61e-4cf7-bd3a-cf4e2ce3031d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:47.913417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:47.914062 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:47.914283 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:47.914557 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:47.915293 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:47.917206 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:47.926027 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:48.151054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1358/2746] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:30:47 2022] POST /identity/v3/users => generated 299 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:30:48.155319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:48.166805 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac9e14c-5ee2-4a93-b9b3-7f15eb084496 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:48.172493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:48.179734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:48.179734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:48.179734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:48.179734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:48.179734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:48.186850 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1389/2747] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:48 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:30:48.191392 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:48.202540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d24c24f-37fd-4a06-b8e3-e0ae23fbf5d0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:48.207617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:48.208527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:48.208813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:48.209096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] PATH_INFO: `/v3/projects/655b8fa3bd3646bc998cf55c66dc60de/users/0fc8ee3c5e6f489a8dd5406e6dff6cc6/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:48.225932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=655b8fa3bd3646bc998cf55c66dc60de, user_id=0fc8ee3c5e6f489a8dd5406e6dff6cc6, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:48.229602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:48.302578 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1359/2748] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:30:48 2022] PUT /identity/v3/projects/655b8fa3bd3646bc998cf55c66dc60de/users/0fc8ee3c5e6f489a8dd5406e6dff6cc6/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:30:48.306554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:48.318733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37137c9-9065-4b85-abcc-4e1ae3d2408f admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:48.379527 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f8138d7-6813-4d19-ae7b-44e00b592160 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:48.380207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8138d7-6813-4d19-ae7b-44e00b592160 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:48.380488 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8138d7-6813-4d19-ae7b-44e00b592160 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:48.380765 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f8138d7-6813-4d19-ae7b-44e00b592160 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:48.381492 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8138d7-6813-4d19-ae7b-44e00b592160 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:48.383228 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f8138d7-6813-4d19-ae7b-44e00b592160 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:48.394196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1390/2749] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:48 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:30:48.398374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:48.409364 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4da33f4b-37eb-43df-a090-821336e41a92 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:48.415613 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:48.416620 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:48.421973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:48.421973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] PATH_INFO: `/v3/projects/655b8fa3bd3646bc998cf55c66dc60de/users/0fc8ee3c5e6f489a8dd5406e6dff6cc6/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:48.421973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=655b8fa3bd3646bc998cf55c66dc60de, user_id=0fc8ee3c5e6f489a8dd5406e6dff6cc6, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:48.422550 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:48.434126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1360/2750] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:30:48 2022] PUT /identity/v3/projects/655b8fa3bd3646bc998cf55c66dc60de/users/0fc8ee3c5e6f489a8dd5406e6dff6cc6/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:48.444428 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082c79c-226a-4f06-9baf-5aa54b6e2652 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:48.445323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082c79c-226a-4f06-9baf-5aa54b6e2652 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:48.445323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7082c79c-226a-4f06-9baf-5aa54b6e2652 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:48.472679 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-7082c79c-226a-4f06-9baf-5aa54b6e2652 None None] MFA Rules not processed for user `0fc8ee3c5e6f489a8dd5406e6dff6cc6`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:48.473961 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-7082c79c-226a-4f06-9baf-5aa54b6e2652 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:48.540618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:48.544551 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1391/2751] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:48 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:48.555916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:48.557880 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:48.618940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e64056-32d8-4899-8f8b-2376dc4b45c0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:48.623385 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740a13a3-ef11-4cbb-a2dc-fa1c0821a417 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:48.624068 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740a13a3-ef11-4cbb-a2dc-fa1c0821a417 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:48.624407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740a13a3-ef11-4cbb-a2dc-fa1c0821a417 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:48.624694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740a13a3-ef11-4cbb-a2dc-fa1c0821a417 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:48.634539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740a13a3-ef11-4cbb-a2dc-fa1c0821a417 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:48.636083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740a13a3-ef11-4cbb-a2dc-fa1c0821a417 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:48.647139 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1361/2752] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:48 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:30:48.676196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082c79c-226a-4f06-9baf-5aa54b6e2652 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:48.747952 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7082c79c-226a-4f06-9baf-5aa54b6e2652 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:48.752605 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:48.753478 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:48.753759 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:48.754037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] PATH_INFO: `/v3/users/c8387cc5880c491a82875e53ecd2aa43` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:48.755362 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8387cc5880c491a82875e53ecd2aa43)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:48.757295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:48.961494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8387cc5880c491a82875e53ecd2aa43'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.052577 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8387cc5880c491a82875e53ecd2aa43'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.061030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8387cc5880c491a82875e53ecd2aa43'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.068639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8387cc5880c491a82875e53ecd2aa43'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.068639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8387cc5880c491a82875e53ecd2aa43'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.081500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1392/2753] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:30:48 2022] DELETE /identity/v3/users/c8387cc5880c491a82875e53ecd2aa43 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:49.086075 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:49.086564 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:49.086677 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:49.105262 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:49.106563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:49.168250 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1362/2754] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:49.178184 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:49.179767 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:49.239720 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd6e05a6-1c1a-4cff-82cb-e01e04b75bc6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:49.243327 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:49.243979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:49.244261 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:49.244536 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:49.254808 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:49.256270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:49.267238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1393/2755] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:49 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 14 12:30:49.703134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:49.704655 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:49.729995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4105a99a-637f-454d-bc74-df6c2e9eb03d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:49.733587 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:49.734457 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:49.734739 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:49.735140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] PATH_INFO: `/v3/projects/84a6ecf0fc1b48ebb7f45e4b657859fc` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:49.736208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=84a6ecf0fc1b48ebb7f45e4b657859fc)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:49.738203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:49.857477 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84a6ecf0fc1b48ebb7f45e4b657859fc', 'request_id': 'req-21df9862-9c85-4794-adab-6696544647f7'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.859554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:INSERT=4,keystone:SELECT=91,keystone:DELETE=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:49.863741 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84a6ecf0fc1b48ebb7f45e4b657859fc', 'request_id': 'req-21df9862-9c85-4794-adab-6696544647f7'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.927411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84a6ecf0fc1b48ebb7f45e4b657859fc', 'request_id': 'req-21df9862-9c85-4794-adab-6696544647f7'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.939482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] The token cache is being invalidate because project 84a6ecf0fc1b48ebb7f45e4b657859fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:30:49.943277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.943277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:49.943277 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1363/2756] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:30:49 2022] DELETE /identity/v3/projects/84a6ecf0fc1b48ebb7f45e4b657859fc => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:30:49.946249 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:49.957476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f8e8a1e-4318-4a2a-adc0-5be8e9d01302 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:49.976836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:50.017970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:50.018890 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:50.019171 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:50.019443 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] PATH_INFO: `/v3/users/0fc8ee3c5e6f489a8dd5406e6dff6cc6` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:50.020654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0fc8ee3c5e6f489a8dd5406e6dff6cc6)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:50.022563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:50.217319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0fc8ee3c5e6f489a8dd5406e6dff6cc6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:50.310010 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0fc8ee3c5e6f489a8dd5406e6dff6cc6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:50.317863 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0fc8ee3c5e6f489a8dd5406e6dff6cc6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:50.318718 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0fc8ee3c5e6f489a8dd5406e6dff6cc6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:50.319374 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-d75f41ae-a82e-40bc-9ae6-0cf3a4ba2f26 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0fc8ee3c5e6f489a8dd5406e6dff6cc6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:50.338417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1394/2757] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:30:49 2022] DELETE /identity/v3/users/0fc8ee3c5e6f489a8dd5406e6dff6cc6 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:50.788155 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:50.790272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:50.803854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21df9862-9c85-4794-adab-6696544647f7 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:50.987485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:50.988360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:50.988648 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:50.988980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] PATH_INFO: `/v3/projects/655b8fa3bd3646bc998cf55c66dc60de` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:50.990797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] RBAC: Authorizing `identity:delete_project(project_id=655b8fa3bd3646bc998cf55c66dc60de)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:50.992458 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:51.102361 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '655b8fa3bd3646bc998cf55c66dc60de', 'request_id': 'req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:51.108122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '655b8fa3bd3646bc998cf55c66dc60de', 'request_id': 'req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:51.172692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '655b8fa3bd3646bc998cf55c66dc60de', 'request_id': 'req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:51.184938 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] The token cache is being invalidate because project 655b8fa3bd3646bc998cf55c66dc60de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:30:51.185228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:51.185912 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-bcff354f-f6d7-4589-8c0b-7bfa8c62a67e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:30:51.187499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1364/2758] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:30:50 2022] DELETE /identity/v3/projects/655b8fa3bd3646bc998cf55c66dc60de => generated 0 bytes in 400 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:51.192416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cd3d45-172e-427b-9900-59ef2e0dbf2e None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.192901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cd3d45-172e-427b-9900-59ef2e0dbf2e None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.193015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cd3d45-172e-427b-9900-59ef2e0dbf2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.212318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-37cd3d45-172e-427b-9900-59ef2e0dbf2e None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:51.213563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-37cd3d45-172e-427b-9900-59ef2e0dbf2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:51.278145 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1395/2759] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:30:51.287596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89ca6e2-bce4-4e3d-b13b-d585df14fd58 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.287957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89ca6e2-bce4-4e3d-b13b-d585df14fd58 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.288235 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f89ca6e2-bce4-4e3d-b13b-d585df14fd58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.309184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-f89ca6e2-bce4-4e3d-b13b-d585df14fd58 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:51.310394 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f89ca6e2-bce4-4e3d-b13b-d585df14fd58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:51.318204 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1365/2760] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:30:51.322473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cd3d45-172e-427b-9900-59ef2e0dbf2e None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:51.332413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cd3d45-172e-427b-9900-59ef2e0dbf2e None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:51.336857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:51.337533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.337814 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.338084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.338829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:30:51.339136 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:51.341326 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:51.352930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1396/2761] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:30:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:30:51.357974 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89ca6e2-bce4-4e3d-b13b-d585df14fd58 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:51.367553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f89ca6e2-bce4-4e3d-b13b-d585df14fd58 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:51.380200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:51.380863 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.381151 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.381429 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.382131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:51.384175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:51.473960 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1366/2762] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:30:51 2022] POST /identity/v3/projects => generated 373 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:30:51.478162 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:51.489674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3193e52-ffda-42d9-bbd7-e52ea539652e admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:51.515677 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:51.549150 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:51.549819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.550099 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.550485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.551203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:51.553124 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:51.658699 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1397/2763] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:30:51 2022] POST /identity/v3/users => generated 287 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:30:51.659766 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:51.671352 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870d6b73-6546-40eb-b664-b58cc4b6a36f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:51.676918 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:51.677651 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.677996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.678287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.679029 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:51.680696 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:51.692042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1367/2764] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:51 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:30:51.696131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:51.706575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-420f1c17-b02a-4737-b1a9-106d33d1f05c admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:51.711359 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1818ca1b-c180-47da-b6c5-b287a928720d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:51.712020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818ca1b-c180-47da-b6c5-b287a928720d admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.712297 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818ca1b-c180-47da-b6c5-b287a928720d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.712570 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1818ca1b-c180-47da-b6c5-b287a928720d admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.713290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1818ca1b-c180-47da-b6c5-b287a928720d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:51.714932 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1818ca1b-c180-47da-b6c5-b287a928720d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:51.725845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1398/2765] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:51 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:30:51.730133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:51.740859 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ae1242-7ffb-44ff-a6ee-4cac5e20653a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:51.745797 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:51.747508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.747781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.748051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] PATH_INFO: `/v3/projects/f0510d496daf4bc9a2f75c2d0f5a3507/users/bee54bd04b8245858adf54f744dedd06/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:51.763124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0510d496daf4bc9a2f75c2d0f5a3507, user_id=bee54bd04b8245858adf54f744dedd06, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:51.765786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:51.852796 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1368/2766] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:30:51 2022] PUT /identity/v3/projects/f0510d496daf4bc9a2f75c2d0f5a3507/users/bee54bd04b8245858adf54f744dedd06/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:51.858529 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df99c3b-210a-4bd9-877f-7c840f87d518 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:51.858843 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df99c3b-210a-4bd9-877f-7c840f87d518 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:51.859121 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df99c3b-210a-4bd9-877f-7c840f87d518 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:52.000925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-0df99c3b-210a-4bd9-877f-7c840f87d518 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:52.002246 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-0df99c3b-210a-4bd9-877f-7c840f87d518 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:52.063733 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1399/2767] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:52.073482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:52.075478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:52.085423 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:52.134560 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84949e3c-be3f-4652-a0ba-ab18e44a7635 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:52.139131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:52.139848 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:52.140198 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:52.140540 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:52.149793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:52.151472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:52.161234 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1369/2768] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:52 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:30:53.313139 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b067ce-c788-45ed-8c41-23501da9a9d7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:53.313721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b067ce-c788-45ed-8c41-23501da9a9d7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:53.313721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88b067ce-c788-45ed-8c41-23501da9a9d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:53.340847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-88b067ce-c788-45ed-8c41-23501da9a9d7 None None] MFA Rules not processed for user `bee54bd04b8245858adf54f744dedd06`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:53.342078 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-88b067ce-c788-45ed-8c41-23501da9a9d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:53.410091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1400/2769] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:53 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:53.414555 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:53.425144 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad0e6ccf-4eb2-4366-bcd8-4431cb8b7a7a service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:53.429794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:53.430501 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:53.430811 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:53.431103 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:53.431809 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:53.433765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:53.521922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1370/2770] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:30:53 2022] POST /identity/v3/projects => generated 371 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:30:53.526012 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b067ce-c788-45ed-8c41-23501da9a9d7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:53.537022 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88b067ce-c788-45ed-8c41-23501da9a9d7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:53.570845 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:53.593876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:53.594545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:53.600345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:53.600345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:53.600345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:53.600345 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:53.690400 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1401/2771] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:30:53 2022] POST /identity/v3/users => generated 292 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:30:53.694317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:53.705222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56171bea-1334-4f7f-b18f-f55941966c0e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:53.709817 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:53.710527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:53.710846 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:53.711157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:53.711899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:53.713512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:53.724768 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1371/2772] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:53 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:30:53.728928 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:53.738939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89cf47a8-15fd-4382-b43b-18cf9c2d02df admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:53.743791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:53.744545 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:53.744828 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:53.745103 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] PATH_INFO: `/v3/projects/aa770184ba2d4cdea8f31cdc392b8441/users/02a15e93dbd64620af3c77307703d75d/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:53.760445 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa770184ba2d4cdea8f31cdc392b8441, user_id=02a15e93dbd64620af3c77307703d75d, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:53.764006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:53.878728 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1402/2773] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:30:53 2022] PUT /identity/v3/projects/aa770184ba2d4cdea8f31cdc392b8441/users/02a15e93dbd64620af3c77307703d75d/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:30:53.882642 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:53.893343 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff2ab74-3c3c-4dea-9d85-5ed41d474100 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:53.948407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eefffb59-0205-4f9d-a3d4-9507d50ade85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:53.949093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefffb59-0205-4f9d-a3d4-9507d50ade85 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:53.949376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefffb59-0205-4f9d-a3d4-9507d50ade85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:53.949654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eefffb59-0205-4f9d-a3d4-9507d50ade85 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:53.951127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefffb59-0205-4f9d-a3d4-9507d50ade85 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:53.952734 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eefffb59-0205-4f9d-a3d4-9507d50ade85 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:53.962988 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1372/2774] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:30:53 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:30:53.967537 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:53.977616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee517915-6a24-44f5-b8f5-e549b45c6088 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:53.982348 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:53.984331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:53.984331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:53.984331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] PATH_INFO: `/v3/projects/aa770184ba2d4cdea8f31cdc392b8441/users/02a15e93dbd64620af3c77307703d75d/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:53.985960 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa770184ba2d4cdea8f31cdc392b8441, user_id=02a15e93dbd64620af3c77307703d75d, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:53.988639 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:54.000096 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1403/2775] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:30:53 2022] PUT /identity/v3/projects/aa770184ba2d4cdea8f31cdc392b8441/users/02a15e93dbd64620af3c77307703d75d/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:30:55.304710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:55.305239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:55.305239 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:55.333504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] MFA Rules not processed for user `02a15e93dbd64620af3c77307703d75d`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:30:55.335047 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:55.401078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:30:55.406504 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1373/2776] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:30:55 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:30:55.418195 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:55.419997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:55.479695 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb915a99-fad5-40a6-974e-4bc0d5f45665 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:55.484271 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:55.484954 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:55.485232 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:55.485504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:55.540439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:55.542084 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:55.552318 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1404/2777] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:55 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:30:57.222150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:30:57.223615 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:30:57.235513 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06fab59e-5218-4d9e-9571-7f49bb6d79dc None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:30:57.239030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:30:57.239687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:30:57.239971 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:30:57.240251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:30:57.249121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:30:57.250547 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:30:57.259934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1374/2778] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:30:57 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:05.548264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=75,keystone:INSERT=5 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:07.255742 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=7 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:11.866513 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:11.870584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:11.937736 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2d52bc-a089-4f06-b4dc-6bb84ece3f9d service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:11.943277 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:11.943930 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:11.944212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:11.944490 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:11.946255 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:11.957533 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:11.959148 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:11.968876 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1405/2779] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:11 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:12.618228 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:12.681319 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4fe3709-2e17-4449-8810-e35295ed7486 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:12.685747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:12.686612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:12.686934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:12.687215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] PATH_INFO: `/v3/users/bee54bd04b8245858adf54f744dedd06` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:12.688397 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bee54bd04b8245858adf54f744dedd06)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:12.691320 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:12.902023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bee54bd04b8245858adf54f744dedd06'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:12.966211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bee54bd04b8245858adf54f744dedd06'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:12.966860 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bee54bd04b8245858adf54f744dedd06'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:12.973986 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bee54bd04b8245858adf54f744dedd06'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:12.977594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bee54bd04b8245858adf54f744dedd06'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:12.992281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1375/2780] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:12 2022] DELETE /identity/v3/users/bee54bd04b8245858adf54f744dedd06 => generated 0 bytes in 375 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:13.247974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:13.260803 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-976692f2-30e2-4061-9240-73db472ae757 service placement] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:13.318325 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:13.319235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:13.319518 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:13.319798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] PATH_INFO: `/v3/projects/f0510d496daf4bc9a2f75c2d0f5a3507` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:13.320979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0510d496daf4bc9a2f75c2d0f5a3507)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:13.322967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:13.460681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0510d496daf4bc9a2f75c2d0f5a3507', 'request_id': 'req-224dae74-a5cf-42e6-b12e-a2a439fb811d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.512225 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0510d496daf4bc9a2f75c2d0f5a3507', 'request_id': 'req-224dae74-a5cf-42e6-b12e-a2a439fb811d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.519217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0510d496daf4bc9a2f75c2d0f5a3507', 'request_id': 'req-224dae74-a5cf-42e6-b12e-a2a439fb811d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.529623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] The token cache is being invalidate because project f0510d496daf4bc9a2f75c2d0f5a3507 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:13.529914 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.530554 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.532127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1406/2781] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:13 2022] DELETE /identity/v3/projects/f0510d496daf4bc9a2f75c2d0f5a3507 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:13.536926 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:13.538330 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:13.551563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c440f463-3c46-476a-b7cd-fae14e537101 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:13.606802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:13.607666 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:13.607945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:13.608215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] PATH_INFO: `/v3/users/02a15e93dbd64620af3c77307703d75d` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:13.609407 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02a15e93dbd64620af3c77307703d75d)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:13.611376 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:13.632602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:13.819236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02a15e93dbd64620af3c77307703d75d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.894646 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02a15e93dbd64620af3c77307703d75d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.895308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02a15e93dbd64620af3c77307703d75d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.902406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02a15e93dbd64620af3c77307703d75d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.903061 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9f9c17d3-1ff6-447c-a4f3-369458f3a5c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02a15e93dbd64620af3c77307703d75d'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:13.920708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1376/2782] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:13 2022] DELETE /identity/v3/users/02a15e93dbd64620af3c77307703d75d => generated 0 bytes in 384 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:14.170818 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:14.172260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:14.185152 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224dae74-a5cf-42e6-b12e-a2a439fb811d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:14.207963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:14.249301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:14.250151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.250430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.250703 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] PATH_INFO: `/v3/projects/aa770184ba2d4cdea8f31cdc392b8441` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.252550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa770184ba2d4cdea8f31cdc392b8441)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:14.254430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:14.366442 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa770184ba2d4cdea8f31cdc392b8441', 'request_id': 'req-edf456cf-258d-4e2a-af2b-0752c1bcb49e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:14.423088 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa770184ba2d4cdea8f31cdc392b8441', 'request_id': 'req-edf456cf-258d-4e2a-af2b-0752c1bcb49e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:14.429926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa770184ba2d4cdea8f31cdc392b8441', 'request_id': 'req-edf456cf-258d-4e2a-af2b-0752c1bcb49e'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:14.440659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] The token cache is being invalidate because project aa770184ba2d4cdea8f31cdc392b8441 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:14.440951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:14.441592 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-edf456cf-258d-4e2a-af2b-0752c1bcb49e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:14.443168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1407/2783] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:14 2022] DELETE /identity/v3/projects/aa770184ba2d4cdea8f31cdc392b8441 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:14.449579 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e29053d-730b-41c8-9220-3b99f76a58fe None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.449868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e29053d-730b-41c8-9220-3b99f76a58fe None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.450145 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e29053d-730b-41c8-9220-3b99f76a58fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.467770 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-8e29053d-730b-41c8-9220-3b99f76a58fe None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:14.468997 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8e29053d-730b-41c8-9220-3b99f76a58fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:14.527542 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1377/2784] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:14.537423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02656e8-c09e-4ad4-b8c1-2c7c739f1b9c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.537715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02656e8-c09e-4ad4-b8c1-2c7c739f1b9c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.537994 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d02656e8-c09e-4ad4-b8c1-2c7c739f1b9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.556593 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-d02656e8-c09e-4ad4-b8c1-2c7c739f1b9c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:14.557798 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-d02656e8-c09e-4ad4-b8c1-2c7c739f1b9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:14.566210 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1408/2785] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:14.570428 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e29053d-730b-41c8-9220-3b99f76a58fe None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:14.580124 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e29053d-730b-41c8-9220-3b99f76a58fe None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:14.584601 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:14.585242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.585521 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.585793 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.586545 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:31:14.587260 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:14.589624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:14.601661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1378/2786] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:31:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:31:14.605821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02656e8-c09e-4ad4-b8c1-2c7c739f1b9c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:14.615552 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d02656e8-c09e-4ad4-b8c1-2c7c739f1b9c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:14.620109 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:14.620749 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.621028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.621302 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.621977 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:14.623894 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:14.719111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1409/2787] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:14 2022] POST /identity/v3/projects => generated 373 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:31:14.720837 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:14.731270 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d953033-f0af-4809-a82e-639c3b80fbfb admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:14.788775 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:14.789441 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.789726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.790004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.790728 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:14.792922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:14.927402 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1379/2788] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:14 2022] POST /identity/v3/users => generated 287 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:14.931522 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:14.941663 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d12a09-73df-4ffc-83b4-2fbe109f13a9 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:14.946091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:14.946740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.947049 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.947323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.948091 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:14.949672 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:14.959717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1410/2789] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:14.963499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:14.973834 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1e940f-ce6d-4beb-a2a1-c2a2948193ec admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:14.978213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c784c104-b242-4ebf-87e2-d8f49ce21081 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:14.978894 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784c104-b242-4ebf-87e2-d8f49ce21081 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:14.979175 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784c104-b242-4ebf-87e2-d8f49ce21081 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:14.979461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c784c104-b242-4ebf-87e2-d8f49ce21081 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:14.980191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c784c104-b242-4ebf-87e2-d8f49ce21081 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:14.981747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c784c104-b242-4ebf-87e2-d8f49ce21081 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:14.992341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1380/2790] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:14.996287 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.006267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6396b1-c0f8-4bca-b46d-c11463b11da5 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:15.010675 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:15.011564 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.011846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.012119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] PATH_INFO: `/v3/projects/78b6cc9f4ed945eeaf871ae17f553acf/users/ebc39a0ade6a4dafb4d0242f9fbe5ea9/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.026860 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78b6cc9f4ed945eeaf871ae17f553acf, user_id=ebc39a0ade6a4dafb4d0242f9fbe5ea9, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:15.029493 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:15.117563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1411/2791] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:14 2022] PUT /identity/v3/projects/78b6cc9f4ed945eeaf871ae17f553acf/users/ebc39a0ade6a4dafb4d0242f9fbe5ea9/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:15.127765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fd2ddc-00f0-41d6-b323-c936b7f3c9ad None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.128059 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fd2ddc-00f0-41d6-b323-c936b7f3c9ad None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.128344 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38fd2ddc-00f0-41d6-b323-c936b7f3c9ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.134982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:DELETE=7,keystone:SELECT=89,keystone:INSERT=4 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:15.154338 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-38fd2ddc-00f0-41d6-b323-c936b7f3c9ad None None] MFA Rules not processed for user `ebc39a0ade6a4dafb4d0242f9fbe5ea9`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:15.155566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-38fd2ddc-00f0-41d6-b323-c936b7f3c9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:15.218709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1381/2792] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:15 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:15.223732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.269945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:15.284031 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e3a51e-17c3-4a6f-9809-f73468bba957 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:15.288702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:15.289353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.289636 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.289913 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.290591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:15.292506 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:15.387825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1412/2793] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:15 2022] POST /identity/v3/projects => generated 375 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:15.392569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fd2ddc-00f0-41d6-b323-c936b7f3c9ad None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.403439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38fd2ddc-00f0-41d6-b323-c936b7f3c9ad None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:15.459908 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:15.460572 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.460854 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.461131 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.461832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:15.463909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:15.564448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1382/2794] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:15 2022] POST /identity/v3/users => generated 294 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:15.568198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.578656 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa47dcdc-1457-47bf-9c4b-431a641f8922 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:15.584358 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:15.585024 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.585303 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.585575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.586295 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:15.587918 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:15.598887 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1413/2795] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:15.602987 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.612849 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2138fee-2325-4168-af72-017215c0086d admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:15.617456 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:15.618324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.618605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.618916 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] PATH_INFO: `/v3/projects/4c1e3d1218364efaad19bd4f3c3dc803/users/053950704d144e29b415abba6c570c18/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.633765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c1e3d1218364efaad19bd4f3c3dc803, user_id=053950704d144e29b415abba6c570c18, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:15.636431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:15.706515 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1383/2796] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:15 2022] PUT /identity/v3/projects/4c1e3d1218364efaad19bd4f3c3dc803/users/053950704d144e29b415abba6c570c18/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:15.710466 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.721030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d84e2163-6dcc-443c-8df3-76309571ae91 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:15.776354 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ddbb8c-59bb-46d6-94bb-7555d678b499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:15.777082 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ddbb8c-59bb-46d6-94bb-7555d678b499 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.777366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ddbb8c-59bb-46d6-94bb-7555d678b499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.777645 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ddbb8c-59bb-46d6-94bb-7555d678b499 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.778394 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ddbb8c-59bb-46d6-94bb-7555d678b499 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:15.780042 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ddbb8c-59bb-46d6-94bb-7555d678b499 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:15.789755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1414/2797] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:15 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:15.794042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.804851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0594ec-31fc-4b3c-8d6a-4786cca8257c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:15.809421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:15.810290 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.810563 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.810872 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] PATH_INFO: `/v3/projects/4c1e3d1218364efaad19bd4f3c3dc803/users/053950704d144e29b415abba6c570c18/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.812825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c1e3d1218364efaad19bd4f3c3dc803, user_id=053950704d144e29b415abba6c570c18, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:15.815813 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:15.825622 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1384/2798] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:15 2022] PUT /identity/v3/projects/4c1e3d1218364efaad19bd4f3c3dc803/users/053950704d144e29b415abba6c570c18/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:15.887768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25efe94-ec97-4314-847a-7955a79a3c81 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:15.888060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25efe94-ec97-4314-847a-7955a79a3c81 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:15.888342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f25efe94-ec97-4314-847a-7955a79a3c81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:15.915470 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-f25efe94-ec97-4314-847a-7955a79a3c81 None None] MFA Rules not processed for user `053950704d144e29b415abba6c570c18`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:15.916674 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f25efe94-ec97-4314-847a-7955a79a3c81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:15.981892 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1415/2799] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:15 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:15.992711 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:15.994378 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:16.047078 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:16.052488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54bfd243-ff94-474c-b696-f5610128e9d2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:16.058467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef913d5-5334-4bb4-8f32-149b07614dcd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:16.058467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef913d5-5334-4bb4-8f32-149b07614dcd service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:16.058467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef913d5-5334-4bb4-8f32-149b07614dcd service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:16.058467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef913d5-5334-4bb4-8f32-149b07614dcd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:16.113281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef913d5-5334-4bb4-8f32-149b07614dcd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:16.114638 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef913d5-5334-4bb4-8f32-149b07614dcd service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:16.123312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1385/2800] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:15 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:19.155270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25efe94-ec97-4314-847a-7955a79a3c81 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:19.200810 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:19.218267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f25efe94-ec97-4314-847a-7955a79a3c81 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:19.222717 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:19.223666 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:19.223967 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:19.224242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] PATH_INFO: `/v3/users/ebc39a0ade6a4dafb4d0242f9fbe5ea9` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:19.225416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ebc39a0ade6a4dafb4d0242f9fbe5ea9)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:19.227375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:19.391001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ebc39a0ade6a4dafb4d0242f9fbe5ea9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:19.480260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ebc39a0ade6a4dafb4d0242f9fbe5ea9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:19.488193 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ebc39a0ade6a4dafb4d0242f9fbe5ea9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:19.488815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ebc39a0ade6a4dafb4d0242f9fbe5ea9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:19.489413 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ebc39a0ade6a4dafb4d0242f9fbe5ea9'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:19.506962 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1416/2801] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:19 2022] DELETE /identity/v3/users/ebc39a0ade6a4dafb4d0242f9fbe5ea9 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:31:19.511925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:19.512438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:19.512567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:19.529881 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:19.531125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:19.590609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1386/2802] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:19.600405 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:19.601825 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:19.659130 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d3cde96-bb9c-48ff-971c-57df996833c5 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:19.662608 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:19.663275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:19.663558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:19.663832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:19.672791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:19.674732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:19.684175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1417/2803] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:20.130663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:20.132176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:20.144448 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f185031-f019-41b7-8f6c-e528af177a4b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:20.149383 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:20.150917 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:20.151354 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:20.151669 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] PATH_INFO: `/v3/projects/78b6cc9f4ed945eeaf871ae17f553acf` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:20.152890 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] RBAC: Authorizing `identity:delete_project(project_id=78b6cc9f4ed945eeaf871ae17f553acf)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:20.154789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:20.266507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78b6cc9f4ed945eeaf871ae17f553acf', 'request_id': 'req-9d65ec58-5c07-4432-8892-6ad401eabc5e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.272520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78b6cc9f4ed945eeaf871ae17f553acf', 'request_id': 'req-9d65ec58-5c07-4432-8892-6ad401eabc5e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.368746 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78b6cc9f4ed945eeaf871ae17f553acf', 'request_id': 'req-9d65ec58-5c07-4432-8892-6ad401eabc5e'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.381201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] The token cache is being invalidate because project 78b6cc9f4ed945eeaf871ae17f553acf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:20.381489 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.382371 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.383955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1387/2804] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:20 2022] DELETE /identity/v3/projects/78b6cc9f4ed945eeaf871ae17f553acf => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:20.388032 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:20.397788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d131adae-dc84-4779-8b9c-bf3b92ff897a service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:20.454074 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:20.455238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:20.455517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:20.455790 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] PATH_INFO: `/v3/users/053950704d144e29b415abba6c570c18` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:20.456974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] RBAC: Authorizing `identity:delete_user(user_id=053950704d144e29b415abba6c570c18)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:20.458916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:20.463342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:20.708299 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '053950704d144e29b415abba6c570c18'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.803111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '053950704d144e29b415abba6c570c18'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.810487 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '053950704d144e29b415abba6c570c18'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.811131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '053950704d144e29b415abba6c570c18'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.811740 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-76c71755-844d-4b07-934b-2f4bad5f97fe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '053950704d144e29b415abba6c570c18'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:20.828753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1418/2805] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:20 2022] DELETE /identity/v3/users/053950704d144e29b415abba6c570c18 => generated 0 bytes in 441 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:31:21.261661 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:21.278418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:21.278418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d65ec58-5c07-4432-8892-6ad401eabc5e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:21.299070 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:21.336566 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:21.337462 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:21.337747 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:21.338023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] PATH_INFO: `/v3/projects/4c1e3d1218364efaad19bd4f3c3dc803` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:21.339315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c1e3d1218364efaad19bd4f3c3dc803)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:21.341265 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:21.474832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c1e3d1218364efaad19bd4f3c3dc803', 'request_id': 'req-915a9692-f724-4e1f-a2e4-f8a6eafd253b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:21.480365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c1e3d1218364efaad19bd4f3c3dc803', 'request_id': 'req-915a9692-f724-4e1f-a2e4-f8a6eafd253b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:21.594906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c1e3d1218364efaad19bd4f3c3dc803', 'request_id': 'req-915a9692-f724-4e1f-a2e4-f8a6eafd253b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:21.606825 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] The token cache is being invalidate because project 4c1e3d1218364efaad19bd4f3c3dc803 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80488) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:21.607122 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:21.616499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-915a9692-f724-4e1f-a2e4-f8a6eafd253b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:21.616499 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1388/2806] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:21 2022] DELETE /identity/v3/projects/4c1e3d1218364efaad19bd4f3c3dc803 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:21.616756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4685f7f4-9a38-48f2-87b0-f5867d7042a7 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:21.616756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4685f7f4-9a38-48f2-87b0-f5867d7042a7 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:21.616756 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4685f7f4-9a38-48f2-87b0-f5867d7042a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:21.634237 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-4685f7f4-9a38-48f2-87b0-f5867d7042a7 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:21.635494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-4685f7f4-9a38-48f2-87b0-f5867d7042a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:21.695340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1419/2807] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:21.703037 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a51cdd3-6f88-4fb0-8a2e-4e2f87339394 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:21.705663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a51cdd3-6f88-4fb0-8a2e-4e2f87339394 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:21.705663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a51cdd3-6f88-4fb0-8a2e-4e2f87339394 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:21.722879 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-0a51cdd3-6f88-4fb0-8a2e-4e2f87339394 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:21.724125 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0a51cdd3-6f88-4fb0-8a2e-4e2f87339394 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:21.732447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1389/2808] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:21.739063 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4685f7f4-9a38-48f2-87b0-f5867d7042a7 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:21.748517 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4685f7f4-9a38-48f2-87b0-f5867d7042a7 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:21.753700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:21.754346 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:21.754625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:21.754933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] PATH_INFO: `/v3/domains` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:21.755652 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80487) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:31:21.756006 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:21.757591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:21.769197 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1420/2809] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:31:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:31:21.775888 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a51cdd3-6f88-4fb0-8a2e-4e2f87339394 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:21.785345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a51cdd3-6f88-4fb0-8a2e-4e2f87339394 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:21.789968 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:21.790616 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:21.790936 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:21.791218 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:21.791955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:21.794109 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:21.919366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1390/2810] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:21 2022] POST /identity/v3/projects => generated 377 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:21.924046 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:21.934030 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad906eef-b8fe-41d6-b536-a90219574e48 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:21.988794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:21.989451 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:21.989727 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:21.989998 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:21.990685 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:21.992604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.090368 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:22.095439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1421/2811] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:21 2022] POST /identity/v3/users => generated 289 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:31:22.099484 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.109591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7608f4a-5153-470d-8f4d-8c4271756b8d admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.115263 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.115909 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.116187 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.116455 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.117166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.118786 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.129712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1391/2812] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:22 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:22.133646 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.143597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1855a012-6dbc-429f-abd5-b4ae3f2fd746 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.148151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-584cbde5-ea7b-43ed-95ba-beb90287d647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.148833 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cbde5-ea7b-43ed-95ba-beb90287d647 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.149132 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cbde5-ea7b-43ed-95ba-beb90287d647 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.149410 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-584cbde5-ea7b-43ed-95ba-beb90287d647 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.150125 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cbde5-ea7b-43ed-95ba-beb90287d647 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.151743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-584cbde5-ea7b-43ed-95ba-beb90287d647 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.163426 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1422/2813] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:22.172451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.183137 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9bbf6-68f3-4682-9137-0f6fc15dc818 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.187838 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.188853 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.189138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.189418 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] PATH_INFO: `/v3/projects/84c0b2ff48d4484da9d8387b8a4b0e07/users/edba58b502ce4108b6d319b3a436f34c/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.203874 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] RBAC: Authorizing `identity:create_grant(project_id=84c0b2ff48d4484da9d8387b8a4b0e07, user_id=edba58b502ce4108b6d319b3a436f34c, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.206559 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.286474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1392/2814] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:22 2022] PUT /identity/v3/projects/84c0b2ff48d4484da9d8387b8a4b0e07/users/edba58b502ce4108b6d319b3a436f34c/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:22.296815 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54cf1b-ce51-4060-84bf-0a6abec972de None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.297107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54cf1b-ce51-4060-84bf-0a6abec972de None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.297386 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf54cf1b-ce51-4060-84bf-0a6abec972de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.324068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-bf54cf1b-ce51-4060-84bf-0a6abec972de None None] MFA Rules not processed for user `edba58b502ce4108b6d319b3a436f34c`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:22.325276 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-bf54cf1b-ce51-4060-84bf-0a6abec972de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:22.387057 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1423/2815] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:22 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:22.391609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.442176 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:22.453114 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ed2bdd-cbad-4d25-a82f-3c6b3dfc472c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.457952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.458612 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.458925 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.459201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.459886 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.461799 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.575592 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1393/2816] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:22 2022] POST /identity/v3/projects => generated 377 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:22.580198 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54cf1b-ce51-4060-84bf-0a6abec972de None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.590958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf54cf1b-ce51-4060-84bf-0a6abec972de None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.647026 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.647701 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.648017 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.648519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.649397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.651331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.803467 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1424/2817] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:22 2022] POST /identity/v3/users => generated 289 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:22.807541 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.817769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52b6d75-e60c-44ba-8c26-3ea797cfb937 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.822236 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.822932 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.823214 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.823490 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.824220 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.825803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.836553 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1394/2818] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:22.840475 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.851090 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-396b2fd0-a673-491d-9d13-be09ca184218 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.855622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab95ce2c-eac9-4521-82f0-1602c6ee2064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.856275 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab95ce2c-eac9-4521-82f0-1602c6ee2064 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.856556 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab95ce2c-eac9-4521-82f0-1602c6ee2064 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.856831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab95ce2c-eac9-4521-82f0-1602c6ee2064 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.857553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab95ce2c-eac9-4521-82f0-1602c6ee2064 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.859182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab95ce2c-eac9-4521-82f0-1602c6ee2064 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.869060 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1425/2819] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:22.873099 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:22.883431 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b66dc41-dc4e-48f4-a879-6f207b26e2b9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:22.887871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:22.888731 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:22.889010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:22.889285 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] PATH_INFO: `/v3/projects/1b39f1c9c989402da4400fd7d0851f92/users/92301f05fbdc48599c835b1ad387b920/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:22.904345 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b39f1c9c989402da4400fd7d0851f92, user_id=92301f05fbdc48599c835b1ad387b920, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:22.906998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:22.997229 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1395/2820] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:22 2022] PUT /identity/v3/projects/1b39f1c9c989402da4400fd7d0851f92/users/92301f05fbdc48599c835b1ad387b920/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:31:23.009306 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb24b41-055c-4b7a-9638-604bde0cce4c None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.009598 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb24b41-055c-4b7a-9638-604bde0cce4c None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.009883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb24b41-055c-4b7a-9638-604bde0cce4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.037217 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-1bb24b41-055c-4b7a-9638-604bde0cce4c None None] MFA Rules not processed for user `92301f05fbdc48599c835b1ad387b920`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:23.038419 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-1bb24b41-055c-4b7a-9638-604bde0cce4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:23.081993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:23.100686 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1426/2821] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:23 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:23.105112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:23.165569 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04fc584e-4b83-4a13-8ad5-8fc07f8e05e5 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:23.170434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:23.171116 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.171403 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.171681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] PATH_INFO: `/v3/projects` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.172374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:23.174465 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:23.284183 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1396/2822] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:23 2022] POST /identity/v3/projects => generated 375 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:23.288322 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb24b41-055c-4b7a-9638-604bde0cce4c None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:23.300034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb24b41-055c-4b7a-9638-604bde0cce4c None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:23.357282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:23.357945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.358230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.358507 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] PATH_INFO: `/v3/users` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.359238 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:23.361235 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:23.465690 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1427/2823] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:23 2022] POST /identity/v3/users => generated 294 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:23.469685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:23.480527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44092cf8-87f4-4ab7-acd3-e39f8f67a8bb admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:23.485224 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:23.486004 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.486284 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.486561 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.487329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:23.489007 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:23.500417 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1397/2824] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:23.504992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:23.515034 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1115d1a4-d44d-41ac-ba64-55615b8739ae admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:23.520532 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:23.521417 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.521698 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.521974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] PATH_INFO: `/v3/projects/42dcd6e599f54cdca5b90e1592115a40/users/3954c420f1e5417a9d18c900a1911bc5/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.538068 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] RBAC: Authorizing `identity:create_grant(project_id=42dcd6e599f54cdca5b90e1592115a40, user_id=3954c420f1e5417a9d18c900a1911bc5, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:23.540649 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:23.611158 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1428/2825] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:23 2022] PUT /identity/v3/projects/42dcd6e599f54cdca5b90e1592115a40/users/3954c420f1e5417a9d18c900a1911bc5/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:23.615242 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:23.625297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6721e13-bd41-408d-97e0-9c2cb043941f admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:23.681658 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17ee548e-2cc4-46e1-a751-c39b227f36af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:23.682342 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee548e-2cc4-46e1-a751-c39b227f36af admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.682611 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee548e-2cc4-46e1-a751-c39b227f36af admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.682919 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17ee548e-2cc4-46e1-a751-c39b227f36af admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.683647 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee548e-2cc4-46e1-a751-c39b227f36af admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:23.685227 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17ee548e-2cc4-46e1-a751-c39b227f36af admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:23.695525 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1398/2826] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:23 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:23.699743 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:23.710623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce45aed-ebe6-44f3-b98a-877040673e78 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:23.715281 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:23.716165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.716461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.716732 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] PATH_INFO: `/v3/projects/42dcd6e599f54cdca5b90e1592115a40/users/3954c420f1e5417a9d18c900a1911bc5/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.718730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] RBAC: Authorizing `identity:create_grant(project_id=42dcd6e599f54cdca5b90e1592115a40, user_id=3954c420f1e5417a9d18c900a1911bc5, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:23.721360 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:23.732205 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1429/2827] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:23 2022] PUT /identity/v3/projects/42dcd6e599f54cdca5b90e1592115a40/users/3954c420f1e5417a9d18c900a1911bc5/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:23.742359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.742685 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.742996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.762411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:23.770136 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] MFA Rules not processed for user `3954c420f1e5417a9d18c900a1911bc5`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:23.774518 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:23.849712 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1399/2828] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:23 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:23.861122 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:23.863309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:23.920131 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e2701f-e61e-418d-a958-706c7b4d675b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:23.923788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:23.924424 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:23.924708 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:23.924983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:23.977939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:23.981411 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:23.982886 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:23.992294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1430/2829] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:23 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:26.048914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:26.051174 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:26.062973 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2963b0-2cb3-41c7-8ca0-4c7bdadf6d98 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:26.066509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:26.067177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:26.067467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:26.067855 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:26.123264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:26.124693 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:26.133459 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1400/2830] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:26 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:27.584851 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:27.593364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e17716-f3b2-45fc-93e8-44ae2049e079 service cinder] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:27.596911 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e467dc-d01a-4f51-b1bb-1b9e6a1f5d93 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:27.597543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e467dc-d01a-4f51-b1bb-1b9e6a1f5d93 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:27.597821 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e467dc-d01a-4f51-b1bb-1b9e6a1f5d93 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:27.598094 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e467dc-d01a-4f51-b1bb-1b9e6a1f5d93 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:27.606871 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e467dc-d01a-4f51-b1bb-1b9e6a1f5d93 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:27.608290 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e467dc-d01a-4f51-b1bb-1b9e6a1f5d93 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:27.617573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1431/2831] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:27 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:32.266077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:32.340408 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ba3b570-b202-4b86-9c45-2f6e04c0f9fd service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:32.344835 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:32.345708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:32.345995 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:32.346272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] PATH_INFO: `/v3/users/edba58b502ce4108b6d319b3a436f34c` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:32.347491 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] RBAC: Authorizing `identity:delete_user(user_id=edba58b502ce4108b6d319b3a436f34c)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:32.349467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:32.558607 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:32.567662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'edba58b502ce4108b6d319b3a436f34c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:32.637626 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'edba58b502ce4108b6d319b3a436f34c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:32.638291 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'edba58b502ce4108b6d319b3a436f34c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:32.645795 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'edba58b502ce4108b6d319b3a436f34c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:32.646420 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'edba58b502ce4108b6d319b3a436f34c'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:32.664500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1401/2832] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:32 2022] DELETE /identity/v3/users/edba58b502ce4108b6d319b3a436f34c => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:32.669209 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:32.669794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:32.669794 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:32.687618 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:32.688916 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:32.753604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1432/2833] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:32.763507 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:32.765054 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:32.824425 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cffe46d2-5992-4fbf-b34b-95a7ea84427e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:32.827950 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:32.828594 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:32.828907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:32.829191 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:32.838482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:32.839959 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:32.850051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1402/2834] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:33.261723 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:33.264959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:33.276584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33cd97ed-ae40-454c-8e6a-dd2cb01b1eba None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:33.281611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:33.282499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:33.282776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:33.283857 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] PATH_INFO: `/v3/projects/84c0b2ff48d4484da9d8387b8a4b0e07` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:33.285256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] RBAC: Authorizing `identity:delete_project(project_id=84c0b2ff48d4484da9d8387b8a4b0e07)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:33.287771 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:33.392839 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '84c0b2ff48d4484da9d8387b8a4b0e07', 'request_id': 'req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.468602 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '84c0b2ff48d4484da9d8387b8a4b0e07', 'request_id': 'req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.475107 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '84c0b2ff48d4484da9d8387b8a4b0e07', 'request_id': 'req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.485512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] The token cache is being invalidate because project 84c0b2ff48d4484da9d8387b8a4b0e07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:33.485801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.486448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.487992 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1433/2835] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:33 2022] DELETE /identity/v3/projects/84c0b2ff48d4484da9d8387b8a4b0e07 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:33.492149 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:33.502185 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e3f22a-2cfc-42cb-9d44-2b9cd06e61ad service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:33.559287 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:33.560159 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:33.560442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:33.560723 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] PATH_INFO: `/v3/users/92301f05fbdc48599c835b1ad387b920` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:33.561914 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=92301f05fbdc48599c835b1ad387b920)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:33.563866 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:33.734168 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92301f05fbdc48599c835b1ad387b920'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.808492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92301f05fbdc48599c835b1ad387b920'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.809112 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92301f05fbdc48599c835b1ad387b920'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.813900 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:33.817652 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92301f05fbdc48599c835b1ad387b920'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.818309 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92301f05fbdc48599c835b1ad387b920'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:33.835267 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1403/2836] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:33 2022] DELETE /identity/v3/users/92301f05fbdc48599c835b1ad387b920 => generated 0 bytes in 344 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:34.239801 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:34.241591 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:34.255755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5dad9ae-0900-40a4-ad70-72c68dd5fa98 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:34.311983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:34.312831 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:34.313111 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:34.313385 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] PATH_INFO: `/v3/projects/1b39f1c9c989402da4400fd7d0851f92` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:34.314553 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b39f1c9c989402da4400fd7d0851f92)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:34.316494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:34.323021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:34.444252 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b39f1c9c989402da4400fd7d0851f92', 'request_id': 'req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.531418 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b39f1c9c989402da4400fd7d0851f92', 'request_id': 'req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.537993 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b39f1c9c989402da4400fd7d0851f92', 'request_id': 'req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.548571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] The token cache is being invalidate because project 1b39f1c9c989402da4400fd7d0851f92 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:34.548855 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.549485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.551061 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1434/2837] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:34 2022] DELETE /identity/v3/projects/1b39f1c9c989402da4400fd7d0851f92 => generated 0 bytes in 313 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:34.555709 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:34.557117 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:34.570360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd92dbe0-fef1-4b35-ba5d-5f6a30d288cf admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:34.627851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:34.629024 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:34.629312 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:34.629590 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] PATH_INFO: `/v3/users/3954c420f1e5417a9d18c900a1911bc5` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:34.631624 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3954c420f1e5417a9d18c900a1911bc5)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:34.633554 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:34.824957 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3954c420f1e5417a9d18c900a1911bc5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.908254 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3954c420f1e5417a9d18c900a1911bc5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.908877 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3954c420f1e5417a9d18c900a1911bc5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.916803 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3954c420f1e5417a9d18c900a1911bc5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.917438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-4b0b1e01-5ca1-40d4-863d-932a145c99a0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3954c420f1e5417a9d18c900a1911bc5'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:34.935906 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1404/2838] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:34 2022] DELETE /identity/v3/users/3954c420f1e5417a9d18c900a1911bc5 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:35.293800 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:35.295301 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:35.308421 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbbc79a2-bf1d-4171-94e5-0f8e7d12ebe6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:35.365043 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:35.365820 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:35.366101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:35.366375 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] PATH_INFO: `/v3/projects/42dcd6e599f54cdca5b90e1592115a40` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:35.367573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] RBAC: Authorizing `identity:delete_project(project_id=42dcd6e599f54cdca5b90e1592115a40)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:35.369473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:35.460112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '42dcd6e599f54cdca5b90e1592115a40', 'request_id': 'req-5413bdda-1647-40c4-bfbc-bffb4b532c17'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:35.551141 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '42dcd6e599f54cdca5b90e1592115a40', 'request_id': 'req-5413bdda-1647-40c4-bfbc-bffb4b532c17'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:35.557758 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '42dcd6e599f54cdca5b90e1592115a40', 'request_id': 'req-5413bdda-1647-40c4-bfbc-bffb4b532c17'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:35.569968 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] The token cache is being invalidate because project 42dcd6e599f54cdca5b90e1592115a40 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:35.570280 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:35.571001 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-5413bdda-1647-40c4-bfbc-bffb4b532c17 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:35.572616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1435/2839] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:35 2022] DELETE /identity/v3/projects/42dcd6e599f54cdca5b90e1592115a40 => generated 0 bytes in 280 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:35.577439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8ed0b7-6069-4c1c-b1d6-0b5b08704c5c None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:35.577922 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8ed0b7-6069-4c1c-b1d6-0b5b08704c5c None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:35.578023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8ed0b7-6069-4c1c-b1d6-0b5b08704c5c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:35.597492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ce8ed0b7-6069-4c1c-b1d6-0b5b08704c5c None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:35.598736 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ce8ed0b7-6069-4c1c-b1d6-0b5b08704c5c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:35.646257 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:35.657030 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1405/2840] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:35.665247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4805e65-5b5c-4dad-99ea-aaf23f238d10 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:35.665541 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4805e65-5b5c-4dad-99ea-aaf23f238d10 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:35.665822 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4805e65-5b5c-4dad-99ea-aaf23f238d10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:35.684716 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-a4805e65-5b5c-4dad-99ea-aaf23f238d10 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:35.685963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a4805e65-5b5c-4dad-99ea-aaf23f238d10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:35.694826 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1436/2841] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:35.698994 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8ed0b7-6069-4c1c-b1d6-0b5b08704c5c None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:35.709203 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8ed0b7-6069-4c1c-b1d6-0b5b08704c5c None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:35.714166 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:35.714851 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:35.715140 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:35.715411 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:35.716138 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:31:35.716445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:35.718034 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:35.730028 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1406/2842] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:31:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:31:35.735783 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4805e65-5b5c-4dad-99ea-aaf23f238d10 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:35.745117 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4805e65-5b5c-4dad-99ea-aaf23f238d10 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:35.749715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:35.750353 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:35.750632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:35.750959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:35.751632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:35.753508 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:35.767721 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:35.884753 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1437/2843] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:35 2022] POST /identity/v3/projects => generated 385 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:35.888899 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:35.899546 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed21120-bafe-4e1b-adb7-b47dd358bf6e admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:35.955482 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:35.956154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:35.956434 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:35.956710 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:35.957388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:35.959366 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.048789 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1407/2844] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:35 2022] POST /identity/v3/users => generated 293 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:36.053611 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.064196 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77c13e8-ef2d-421a-8528-8a063060af09 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.069755 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.070489 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.070784 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.071083 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.071806 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.073397 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.084065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1438/2845] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:36.087937 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.097749 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e93b7e8-4e5c-4796-96b2-3f9db1b3223a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.102474 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb7fee8-d184-4206-8633-40f067053189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.103163 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7fee8-d184-4206-8633-40f067053189 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.103447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7fee8-d184-4206-8633-40f067053189 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.103720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb7fee8-d184-4206-8633-40f067053189 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.104447 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb7fee8-d184-4206-8633-40f067053189 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.106018 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb7fee8-d184-4206-8633-40f067053189 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.116509 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1408/2846] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 14 12:31:36.120452 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.131013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53f8dff7-7475-4e0a-a75e-4c0b1f8f2851 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.135730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.136599 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.136877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.137151 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] PATH_INFO: `/v3/projects/4120d235d25c4d77957007b9fc5a2130/users/6a4c99dc5c124e9aa9e1268e66a12d2a/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.153309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4120d235d25c4d77957007b9fc5a2130, user_id=6a4c99dc5c124e9aa9e1268e66a12d2a, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.155904 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.214221 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1439/2847] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:36 2022] PUT /identity/v3/projects/4120d235d25c4d77957007b9fc5a2130/users/6a4c99dc5c124e9aa9e1268e66a12d2a/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:36.224472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52e482-dcd1-434b-a255-7c97976ade1b None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.225226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52e482-dcd1-434b-a255-7c97976ade1b None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.225226 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca52e482-dcd1-434b-a255-7c97976ade1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.252008 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-ca52e482-dcd1-434b-a255-7c97976ade1b None None] MFA Rules not processed for user `6a4c99dc5c124e9aa9e1268e66a12d2a`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:36.253222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ca52e482-dcd1-434b-a255-7c97976ade1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:36.317165 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1409/2848] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:36 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:36.321622 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.381867 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05bd042a-9a98-49c0-8ad4-8025c4781c87 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.386528 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.387211 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.387494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.387770 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.388476 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.390364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.493586 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1440/2849] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:36 2022] POST /identity/v3/projects => generated 385 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:36.497783 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52e482-dcd1-434b-a255-7c97976ade1b None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.508360 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca52e482-dcd1-434b-a255-7c97976ade1b None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.555051 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:36.564549 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.565213 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.565492 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.565765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.566442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.568852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.706605 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1410/2850] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:36 2022] POST /identity/v3/users => generated 293 bytes in 209 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:31:36.711112 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.721242 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-417fae44-7283-40fc-91cd-16c7c8bc0458 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.726448 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.727142 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.727416 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.727683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.728406 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.729989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.740715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1441/2851] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:36.744698 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.754654 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a60431-cc31-4bd1-a0fd-c70600876f88 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.759281 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5bfea85-7df7-4228-b863-d9fb1df075cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.759934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bfea85-7df7-4228-b863-d9fb1df075cb admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.760217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bfea85-7df7-4228-b863-d9fb1df075cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.760493 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5bfea85-7df7-4228-b863-d9fb1df075cb admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.761215 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bfea85-7df7-4228-b863-d9fb1df075cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.762824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5bfea85-7df7-4228-b863-d9fb1df075cb admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.773049 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1411/2852] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:36 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:36.776973 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:36.787430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6cea866-7c42-40f5-8f18-28281fb92b29 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:36.791853 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:36.792988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.793269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.793543 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] PATH_INFO: `/v3/projects/362d6e9d7bfe4e37a85480340acca852/users/1704e7387ade4c06880008d4404d9d7b/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.807700 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=362d6e9d7bfe4e37a85480340acca852, user_id=1704e7387ade4c06880008d4404d9d7b, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:36.810293 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:36.929486 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1442/2853] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:36 2022] PUT /identity/v3/projects/362d6e9d7bfe4e37a85480340acca852/users/1704e7387ade4c06880008d4404d9d7b/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:36.939845 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e338546-b86e-4f1c-a469-e5b80e47b77e None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:36.940134 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e338546-b86e-4f1c-a469-e5b80e47b77e None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:36.940438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e338546-b86e-4f1c-a469-e5b80e47b77e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:36.967369 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-2e338546-b86e-4f1c-a469-e5b80e47b77e None None] MFA Rules not processed for user `1704e7387ade4c06880008d4404d9d7b`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:36.968503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-2e338546-b86e-4f1c-a469-e5b80e47b77e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:37.033365 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1412/2854] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:36 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:37.038338 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:37.099854 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4050b1b9-e7d9-4bd6-8608-10c2374325e6 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:37.104734 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:37.105423 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.105730 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.106033 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.106785 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:37.108816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:37.122628 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:37.229791 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1443/2855] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:37 2022] POST /identity/v3/projects => generated 385 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:31:37.237317 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e338546-b86e-4f1c-a469-e5b80e47b77e None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:37.247905 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e338546-b86e-4f1c-a469-e5b80e47b77e None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:37.310221 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:37.310923 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.311209 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.311488 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.312217 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:37.314132 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:37.437688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1413/2856] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:37 2022] POST /identity/v3/users => generated 299 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:37.441895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:37.453081 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3582cc1-a485-404a-a5b1-e6383686d59b admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:37.457540 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:37.458212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.458524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.458836 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.459654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:37.461253 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:37.471779 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1444/2857] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:37.475933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:37.485940 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-513059ae-596a-4ab5-b73f-c12efd906d21 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:37.491222 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:37.492154 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.492487 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.492818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] PATH_INFO: `/v3/projects/73bc84166aae443b9e2f386f16accd28/users/143ffc905b264f939d88b18306daf554/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.508308 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73bc84166aae443b9e2f386f16accd28, user_id=143ffc905b264f939d88b18306daf554, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:37.511060 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:37.594127 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1414/2858] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:37 2022] PUT /identity/v3/projects/73bc84166aae443b9e2f386f16accd28/users/143ffc905b264f939d88b18306daf554/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:37.598065 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:37.608016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27bf4ff2-e0f3-42dc-ae34-6134bd4d36ce admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:37.663526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caa2ddf6-c4c4-4023-b457-d0e454353da7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:37.664215 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2ddf6-c4c4-4023-b457-d0e454353da7 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.664499 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2ddf6-c4c4-4023-b457-d0e454353da7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.664776 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caa2ddf6-c4c4-4023-b457-d0e454353da7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.665495 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa2ddf6-c4c4-4023-b457-d0e454353da7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:37.667182 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caa2ddf6-c4c4-4023-b457-d0e454353da7 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:37.677550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1445/2859] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:37 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:37.681769 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:37.691904 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51c0855a-b3ca-4b5c-adca-8e2653bf7895 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:37.697333 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:37.698241 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] REQUEST_METHOD: `PUT` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.698523 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.698952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] PATH_INFO: `/v3/projects/73bc84166aae443b9e2f386f16accd28/users/143ffc905b264f939d88b18306daf554/roles/e1d0ea88764c4c89a1159e9b09076352` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.700943 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] RBAC: Authorizing `identity:create_grant(project_id=73bc84166aae443b9e2f386f16accd28, user_id=143ffc905b264f939d88b18306daf554, role_id=e1d0ea88764c4c89a1159e9b09076352)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:37.703598 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:37.713935 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1415/2860] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:37 2022] PUT /identity/v3/projects/73bc84166aae443b9e2f386f16accd28/users/143ffc905b264f939d88b18306daf554/roles/e1d0ea88764c4c89a1159e9b09076352 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:37.724964 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.725250 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.725526 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.752866 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] MFA Rules not processed for user `143ffc905b264f939d88b18306daf554`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:37.753997 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:37.821509 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1446/2861] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:37 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:37.832831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:37.834568 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:37.848157 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:37.892273 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4d0e1fe-e329-4f38-a7f0-8d4cec224380 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:37.895970 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:37.896830 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:37.897216 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:37.897604 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:37.957073 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:37.958985 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:37.968240 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1416/2862] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:37 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:39.346072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:39.346072 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:39.427546 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=100 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:39.432788 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897443fd-ba58-484b-8a3a-ee0f8b6df4b9 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:39.438512 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'da24a7ecdcc249f0820add0404a7856c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['ResellerAdmin', 'member', 'reader', 'admin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:39.441715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:39.441715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:39.441715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:39.441715 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:39.454101 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:39.455746 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:39.468154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1447/2863] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:39 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:39.997244 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:39.998745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:40.010609 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acc5b03d-2e34-473a-a710-e3308077ab58 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:40.014177 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:40.014889 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:40.015169 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:40.015442 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:40.073839 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:40.075324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:40.084996 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1417/2864] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:39 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:41.543958 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:41.554274 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87c96e1-359c-4aad-a365-2825c965bb69 service nova] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:41.557925 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60d89e1a-7833-4094-830a-801d55ba2d21 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:41.558563 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d89e1a-7833-4094-830a-801d55ba2d21 service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:41.558957 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d89e1a-7833-4094-830a-801d55ba2d21 service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:41.559239 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d89e1a-7833-4094-830a-801d55ba2d21 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:41.568519 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d89e1a-7833-4094-830a-801d55ba2d21 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:41.569933 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60d89e1a-7833-4094-830a-801d55ba2d21 service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:41.579264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1448/2865] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:41 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:46.120010 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:46.182223 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a74d50-5fc0-464a-95e6-9cc9838f43f5 service cinder] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:46.186840 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:46.187701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:46.187983 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:46.188264 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] PATH_INFO: `/v3/users/6a4c99dc5c124e9aa9e1268e66a12d2a` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:46.189449 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a4c99dc5c124e9aa9e1268e66a12d2a)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:46.191439 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:46.196531 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=100 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:46.419581 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a4c99dc5c124e9aa9e1268e66a12d2a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:46.522675 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a4c99dc5c124e9aa9e1268e66a12d2a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:46.523246 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a4c99dc5c124e9aa9e1268e66a12d2a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:46.530887 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a4c99dc5c124e9aa9e1268e66a12d2a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:46.531567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a4c99dc5c124e9aa9e1268e66a12d2a'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:46.553556 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1418/2866] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:46 2022] DELETE /identity/v3/users/6a4c99dc5c124e9aa9e1268e66a12d2a => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:46.558256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:46.560711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:46.560711 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:46.577089 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:46.578332 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:46.636707 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1449/2867] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:46.646451 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:46.647982 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:46.705804 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b3abbc-dfe5-4c4d-af6f-5f64fa1983e9 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:46.709419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:46.710064 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:46.710346 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:46.710623 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:46.720096 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:46.721562 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:46.731432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1419/2868] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:47.543300 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:47.544901 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:47.556951 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f69b5b8-c51b-4c5e-b712-a13568c8db5a None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:47.561597 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:47.562469 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:47.562752 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:47.563075 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] PATH_INFO: `/v3/projects/4120d235d25c4d77957007b9fc5a2130` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:47.564981 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] RBAC: Authorizing `identity:delete_project(project_id=4120d235d25c4d77957007b9fc5a2130)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:47.566333 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:47.674642 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4120d235d25c4d77957007b9fc5a2130', 'request_id': 'req-33e12855-7c2b-4dee-bb11-114f0495299d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:47.742909 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4120d235d25c4d77957007b9fc5a2130', 'request_id': 'req-33e12855-7c2b-4dee-bb11-114f0495299d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:47.749824 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4120d235d25c4d77957007b9fc5a2130', 'request_id': 'req-33e12855-7c2b-4dee-bb11-114f0495299d'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:47.762340 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] The token cache is being invalidate because project 4120d235d25c4d77957007b9fc5a2130 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:47.762630 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:47.763334 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:47.765118 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1450/2869] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:47 2022] DELETE /identity/v3/projects/4120d235d25c4d77957007b9fc5a2130 => generated 0 bytes in 222 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:47.768933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:47.782002 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788a726f-00b0-4913-b15a-376706c504fa service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:47.836945 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:47.837740 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:47.838023 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:47.838299 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] PATH_INFO: `/v3/users/1704e7387ade4c06880008d4404d9d7b` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:47.839508 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1704e7387ade4c06880008d4404d9d7b)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:47.841454 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:47.965043 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:48.040748 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1704e7387ade4c06880008d4404d9d7b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:48.111575 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1704e7387ade4c06880008d4404d9d7b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:48.112506 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1704e7387ade4c06880008d4404d9d7b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:48.121202 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1704e7387ade4c06880008d4404d9d7b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:48.122467 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1704e7387ade4c06880008d4404d9d7b'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:48.170729 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1420/2870] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:47 2022] DELETE /identity/v3/users/1704e7387ade4c06880008d4404d9d7b => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:31:48.803460 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:48.804114 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:48.819213 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e12855-7c2b-4dee-bb11-114f0495299d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:48.884817 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:48.885696 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:48.885983 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:48.886256 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] PATH_INFO: `/v3/projects/362d6e9d7bfe4e37a85480340acca852` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:48.887579 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=362d6e9d7bfe4e37a85480340acca852)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:48.889497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:48.891888 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:49.001654 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '362d6e9d7bfe4e37a85480340acca852', 'request_id': 'req-6ff6f1fc-e988-437a-bf4f-760b74338aa0'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.119926 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '362d6e9d7bfe4e37a85480340acca852', 'request_id': 'req-6ff6f1fc-e988-437a-bf4f-760b74338aa0'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.126917 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '362d6e9d7bfe4e37a85480340acca852', 'request_id': 'req-6ff6f1fc-e988-437a-bf4f-760b74338aa0'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.137434 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] The token cache is being invalidate because project 362d6e9d7bfe4e37a85480340acca852 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:49.137725 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.138366 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.139940 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1451/2871] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:48 2022] DELETE /identity/v3/projects/362d6e9d7bfe4e37a85480340acca852 => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:49.143952 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:49.145395 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:49.158133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac1765a1-f0f5-4216-9287-864d7620e768 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:49.215720 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:49.216591 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:49.216873 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:49.217150 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] PATH_INFO: `/v3/users/143ffc905b264f939d88b18306daf554` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:49.218350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=143ffc905b264f939d88b18306daf554)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:49.220315 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:49.418875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '143ffc905b264f939d88b18306daf554'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.571104 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '143ffc905b264f939d88b18306daf554'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.571750 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '143ffc905b264f939d88b18306daf554'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.579297 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '143ffc905b264f939d88b18306daf554'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.579933 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-377328ec-4b1f-436f-9ac6-205120dd7fbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '143ffc905b264f939d88b18306daf554'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:49.597955 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1421/2872] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:31:49 2022] DELETE /identity/v3/users/143ffc905b264f939d88b18306daf554 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:50.068168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:50.070632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:50.085342 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ff6f1fc-e988-437a-bf4f-760b74338aa0 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:50.143402 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:50.144263 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.144548 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.144832 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] PATH_INFO: `/v3/projects/73bc84166aae443b9e2f386f16accd28` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.146119 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73bc84166aae443b9e2f386f16accd28)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:50.148059 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:50.289615 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73bc84166aae443b9e2f386f16accd28', 'request_id': 'req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:50.374561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73bc84166aae443b9e2f386f16accd28', 'request_id': 'req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:50.381665 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73bc84166aae443b9e2f386f16accd28', 'request_id': 'req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:50.391823 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] The token cache is being invalidate because project 73bc84166aae443b9e2f386f16accd28 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:31:50.392110 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:50.392737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-b62c9b82-e5ab-4a63-a292-f1644ee0ed85 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:31:50.394313 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1452/2873] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:31:50 2022] DELETE /identity/v3/projects/73bc84166aae443b9e2f386f16accd28 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:50.399978 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e21e83-caa2-4854-9f43-2c89758dd0db None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.400478 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e21e83-caa2-4854-9f43-2c89758dd0db None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.400596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e21e83-caa2-4854-9f43-2c89758dd0db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.419762 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-e3e21e83-caa2-4854-9f43-2c89758dd0db None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:50.420998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-e3e21e83-caa2-4854-9f43-2c89758dd0db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:50.440419 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=83,keystone:DELETE=15,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:50.480787 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1422/2874] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:31:50.488534 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a11fd01-83e6-42d1-b80e-521834702737 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.488829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a11fd01-83e6-42d1-b80e-521834702737 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.489170 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a11fd01-83e6-42d1-b80e-521834702737 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.508432 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-2a11fd01-83e6-42d1-b80e-521834702737 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:50.509738 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-2a11fd01-83e6-42d1-b80e-521834702737 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:50.518021 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1453/2875] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:31:50.522745 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e21e83-caa2-4854-9f43-2c89758dd0db None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:50.532374 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e21e83-caa2-4854-9f43-2c89758dd0db None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:50.537300 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:50.537962 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.538274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.538754 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.539516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:31:50.539875 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:50.541500 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:50.553643 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1423/2876] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:31:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:31:50.558222 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a11fd01-83e6-42d1-b80e-521834702737 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:50.568178 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a11fd01-83e6-42d1-b80e-521834702737 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:50.572895 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:50.573560 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.573846 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.574177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.574974 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:50.576879 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:50.590919 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:50.655619 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1454/2877] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:31:50 2022] POST /identity/v3/projects => generated 369 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:31:50.659765 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:50.670810 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35c9938-14a5-490e-8348-c5e8945901e5 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:50.732326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:50.733013 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.733295 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.733571 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.734269 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:50.736274 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:50.866199 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1424/2878] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:31:50 2022] POST /identity/v3/users => generated 285 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 14 12:31:50.870877 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:50.882949 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-219356b5-5183-4652-86c3-a7d1a5e8b071 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:50.887816 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:50.888754 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.889127 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.889648 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.890269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:50.894979 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:50.906847 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1455/2879] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:50 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:50.908517 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:50.924687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f616df8-2ad3-430a-bfb5-620c5b301f09 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:50.931432 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab97554-6c60-4887-9849-c29e3a3b4298 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:50.932118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab97554-6c60-4887-9849-c29e3a3b4298 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.932406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab97554-6c60-4887-9849-c29e3a3b4298 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.932688 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab97554-6c60-4887-9849-c29e3a3b4298 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.933567 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab97554-6c60-4887-9849-c29e3a3b4298 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:50.935359 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab97554-6c60-4887-9849-c29e3a3b4298 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:50.951794 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1425/2880] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:31:50 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:31:50.953069 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:50.968975 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94815c6-33df-4b5f-bc29-aa552dc511be admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:50.974055 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:50.975778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:50.975778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:50.975778 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] PATH_INFO: `/v3/projects/67637017e0984c75a84069c2e8617bce/users/048bb5fc7000489ebdadf6f5e0d6ea20/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:50.991939 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67637017e0984c75a84069c2e8617bce, user_id=048bb5fc7000489ebdadf6f5e0d6ea20, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:50.994584 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:51.091681 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1456/2881] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:31:50 2022] PUT /identity/v3/projects/67637017e0984c75a84069c2e8617bce/users/048bb5fc7000489ebdadf6f5e0d6ea20/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:31:51.102329 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:51.102808 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:51.103083 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:51.132208 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] MFA Rules not processed for user `048bb5fc7000489ebdadf6f5e0d6ea20`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:31:51.133357 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:51.199674 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1426/2882] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:31:51 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 14 12:31:51.212037 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:51.213811 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:51.297388 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e8dc62-a6d4-4fe3-9b20-a3bba8155d25 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:51.301154 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '86ace7e877e3484b8853536b77c76885', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:51.302336 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:51.302626 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:51.302934 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:51.311976 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:51.313427 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:51.323133 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1457/2883] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:51 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:51.865792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:51.867247 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:51.888812 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:51.937694 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-869a0481-d801-46ca-858b-6d29ad2e721d None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:51.942681 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:51.943388 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:51.943686 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:51.944021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:51.945832 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:51.958042 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:51.960211 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:51.970527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1427/2884] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:51 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:31:55.484683 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:31:55.486177 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:55.543350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35e2f31-c8c9-44ab-86b9-7a8a89b2971d service glance] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:31:55.546989 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:31:55.547633 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:31:55.547945 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:31:55.548282 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:31:55.550549 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:31:55.602671 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:31:55.614538 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:31:55.618795 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:31:55.626946 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1458/2885] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:31:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:32:02.051516 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=23 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:02.975161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:02.976726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:02.992876 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ca1847c-cac4-4e3e-a574-7b8c3e7a1865 service placement] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:02.997438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:02.998350 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:02.998588 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:02.998929 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] PATH_INFO: `/v3/users/d0adaa092d944202b3279d10e0f94a91` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:03.000153 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d0adaa092d944202b3279d10e0f94a91)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:03.002121 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:03.182781 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd0adaa092d944202b3279d10e0f94a91'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.242617 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd0adaa092d944202b3279d10e0f94a91'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.243251 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd0adaa092d944202b3279d10e0f94a91'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.250641 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd0adaa092d944202b3279d10e0f94a91'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.257934 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd0adaa092d944202b3279d10e0f94a91'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.268512 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1428/2886] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:32:02 2022] DELETE /identity/v3/users/d0adaa092d944202b3279d10e0f94a91 => generated 0 bytes in 294 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:32:03.549363 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:03.561270 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7c5218-e8f0-43cd-87c1-493f275d25e3 service neutron] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:03.625485 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:03.626430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:03.626724 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:03.627028 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] PATH_INFO: `/v3/projects/c0d098ca721b409385a08b91d861b295` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:03.628223 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0d098ca721b409385a08b91d861b295)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:03.630357 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:03.743309 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0d098ca721b409385a08b91d861b295', 'request_id': 'req-544fb717-0198-4066-938a-e69541b40a23'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.819576 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0d098ca721b409385a08b91d861b295', 'request_id': 'req-544fb717-0198-4066-938a-e69541b40a23'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.826446 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0d098ca721b409385a08b91d861b295', 'request_id': 'req-544fb717-0198-4066-938a-e69541b40a23'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.837323 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] The token cache is being invalidate because project c0d098ca721b409385a08b91d861b295 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:32:03.837610 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.838266 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:03.839829 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1459/2887] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:32:03 2022] DELETE /identity/v3/projects/c0d098ca721b409385a08b91d861b295 => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:32:03.844005 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:03.845398 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:03.859503 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-764dbece-2ec7-4fe1-a677-18af120b26d2 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:03.916602 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:03.917473 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:03.917755 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:03.918314 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] PATH_INFO: `/v3/users/1738f47382234337901606d8ae00e910` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:03.919544 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1738f47382234337901606d8ae00e910)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:03.921498 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:04.159701 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1738f47382234337901606d8ae00e910'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.236931 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1738f47382234337901606d8ae00e910'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.237539 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1738f47382234337901606d8ae00e910'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.245079 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1738f47382234337901606d8ae00e910'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.245726 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1738f47382234337901606d8ae00e910'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.253737 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:04.277031 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1429/2888] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:32:03 2022] DELETE /identity/v3/users/1738f47382234337901606d8ae00e910 => generated 0 bytes in 433 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:32:04.529473 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:04.530963 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:04.544168 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544fb717-0198-4066-938a-e69541b40a23 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:04.602789 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:04.603659 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:04.603938 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:04.604212 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] PATH_INFO: `/v3/projects/7c864497be59453182efbe81fdf948cb` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:04.605382 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7c864497be59453182efbe81fdf948cb)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:04.607320 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:04.632948 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:04.737504 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7c864497be59453182efbe81fdf948cb', 'request_id': 'req-34411ae6-4139-4317-9693-7c60a77cddd8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.807781 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7c864497be59453182efbe81fdf948cb', 'request_id': 'req-34411ae6-4139-4317-9693-7c60a77cddd8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.814265 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7c864497be59453182efbe81fdf948cb', 'request_id': 'req-34411ae6-4139-4317-9693-7c60a77cddd8'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.825329 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] The token cache is being invalidate because project 7c864497be59453182efbe81fdf948cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:32:04.825616 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.826247 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-34411ae6-4139-4317-9693-7c60a77cddd8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:04.827812 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1460/2889] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:32:04 2022] DELETE /identity/v3/projects/7c864497be59453182efbe81fdf948cb => generated 0 bytes in 299 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:32:14.362466 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=3,keystone:DELETE=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:14.822364 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=15,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:21.751126 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:21.752583 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:21.765472 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a63327-25e7-4736-9620-323ce7c4b49a admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:21.821332 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:21.822196 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:21.822480 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:21.822757 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] PATH_INFO: `/v3/users/048bb5fc7000489ebdadf6f5e0d6ea20` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:21.823980 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] RBAC: Authorizing `identity:delete_user(user_id=048bb5fc7000489ebdadf6f5e0d6ea20)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:21.825907 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:21.992197 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '048bb5fc7000489ebdadf6f5e0d6ea20'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.061824 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '048bb5fc7000489ebdadf6f5e0d6ea20'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.062445 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '048bb5fc7000489ebdadf6f5e0d6ea20'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.070146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '048bb5fc7000489ebdadf6f5e0d6ea20'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.070792 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '048bb5fc7000489ebdadf6f5e0d6ea20'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.088520 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1430/2890] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:32:21 2022] DELETE /identity/v3/users/048bb5fc7000489ebdadf6f5e0d6ea20 => generated 0 bytes in 338 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:32:22.093025 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:22.093524 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:22.093643 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:22.110625 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:32:22.114294 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:22.168706 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1461/2891] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:32:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:32:22.178288 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:22.179727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:22.237133 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6678d4-6b55-44b9-8420-7a7243de455c admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:22.240819 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-276ab8e2-0a89-471f-83d2-7ff73d48aefc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:22.241461 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276ab8e2-0a89-471f-83d2-7ff73d48aefc service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:22.241753 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276ab8e2-0a89-471f-83d2-7ff73d48aefc service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:22.242032 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-276ab8e2-0a89-471f-83d2-7ff73d48aefc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:22.251347 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276ab8e2-0a89-471f-83d2-7ff73d48aefc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:22.252752 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-276ab8e2-0a89-471f-83d2-7ff73d48aefc service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:22.262580 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1431/2892] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:32:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:32:22.667102 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:22.669903 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:22.681264 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29918eb4-a9f8-47f2-a75d-3f830631ab17 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:22.686500 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:22.687430 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:22.687712 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:22.688002 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] PATH_INFO: `/v3/projects/67637017e0984c75a84069c2e8617bce` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:22.689165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] RBAC: Authorizing `identity:delete_project(project_id=67637017e0984c75a84069c2e8617bce)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:22.691105 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:22.803444 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67637017e0984c75a84069c2e8617bce', 'request_id': 'req-6fb9fb0d-83d1-49ba-b753-37f06b493552'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.882269 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67637017e0984c75a84069c2e8617bce', 'request_id': 'req-6fb9fb0d-83d1-49ba-b753-37f06b493552'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.889027 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67637017e0984c75a84069c2e8617bce', 'request_id': 'req-6fb9fb0d-83d1-49ba-b753-37f06b493552'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.899284 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] The token cache is being invalidate because project 67637017e0984c75a84069c2e8617bce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:32:22.899575 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.900207 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-6fb9fb0d-83d1-49ba-b753-37f06b493552 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:22.901764 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1462/2893] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:32:22 2022] DELETE /identity/v3/projects/67637017e0984c75a84069c2e8617bce => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 14 12:32:22.907210 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18e008-d18c-42ef-9274-f1d88a158e81 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:22.907708 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18e008-d18c-42ef-9274-f1d88a158e81 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:22.907831 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a18e008-d18c-42ef-9274-f1d88a158e81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:22.925871 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-5a18e008-d18c-42ef-9274-f1d88a158e81 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:32:22.927660 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-5a18e008-d18c-42ef-9274-f1d88a158e81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:22.944426 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:22.986172 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1432/2894] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:32:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:32:23.001175 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86baaab-d80d-410b-bb86-a8c8943c80a2 None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.001459 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86baaab-d80d-410b-bb86-a8c8943c80a2 None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.001737 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e86baaab-d80d-410b-bb86-a8c8943c80a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.020020 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-e86baaab-d80d-410b-bb86-a8c8943c80a2 None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:32:23.021260 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-e86baaab-d80d-410b-bb86-a8c8943c80a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:23.029331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1463/2895] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:32:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:32:23.033586 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a18e008-d18c-42ef-9274-f1d88a158e81 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:23.044903 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a18e008-d18c-42ef-9274-f1d88a158e81 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:23.056438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:23.056438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.056438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.056438 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] PATH_INFO: `/v3/domains` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.056896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80488) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 14 12:32:23.056896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:23.056896 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:23.065802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1433/2896] 10.222.0.27 () {58 vars in 1152 bytes} [Thu Jul 14 12:32:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 14 12:32:23.070165 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86baaab-d80d-410b-bb86-a8c8943c80a2 None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:23.079283 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e86baaab-d80d-410b-bb86-a8c8943c80a2 None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:23.084539 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:23.085189 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.085494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.085750 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] PATH_INFO: `/v3/projects` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.086438 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:23.088378 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:23.166573 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1464/2897] 10.222.0.27 () {60 vars in 1152 bytes} [Thu Jul 14 12:32:23 2022] POST /identity/v3/projects => generated 375 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:32:23.170956 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:23.181025 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89071a7b-7646-4f9b-a93a-a780c3a7d495 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:23.236852 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:23.237522 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.237921 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.238207 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] PATH_INFO: `/v3/users` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.238924 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:23.240946 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:23.328341 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1434/2898] 10.222.0.27 () {60 vars in 1143 bytes} [Thu Jul 14 12:32:23 2022] POST /identity/v3/users => generated 288 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 14 12:32:23.332558 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:23.343497 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18360aea-6aba-4aad-be3d-0bf9dfa7ee53 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:23.348015 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:23.348669 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.349655 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.349970 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] PATH_INFO: `/v3/roles` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.350702 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:23.352319 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:23.362632 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1465/2899] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:32:23 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:32:23.366527 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:23.376858 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e343b3-8933-4de3-b391-57e35b32f6c8 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:23.381717 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c30c0c8-b1a2-4139-81eb-81e9765049f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:23.382377 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c30c0c8-b1a2-4139-81eb-81e9765049f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.382662 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c30c0c8-b1a2-4139-81eb-81e9765049f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.382976 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c30c0c8-b1a2-4139-81eb-81e9765049f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.383868 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c30c0c8-b1a2-4139-81eb-81e9765049f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:23.385475 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c30c0c8-b1a2-4139-81eb-81e9765049f4 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:23.395533 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1435/2900] 10.222.0.27 () {58 vars in 1121 bytes} [Thu Jul 14 12:32:23 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 14 12:32:23.401016 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:23.411126 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0b1fd-dacc-411d-9eae-2e7c3f782974 admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:23.416458 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:23.417331 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] REQUEST_METHOD: `PUT` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.417612 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.417905 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] PATH_INFO: `/v3/projects/dd62f55e2c734a4f8a8c38fa009b157b/users/51bdc8d3518248059037f94067905d18/roles/56f2092f7f5e42b09679dec166d174c5` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.432819 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] RBAC: Authorizing `identity:create_grant(project_id=dd62f55e2c734a4f8a8c38fa009b157b, user_id=51bdc8d3518248059037f94067905d18, role_id=56f2092f7f5e42b09679dec166d174c5)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:23.435550 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:23.493387 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1466/2901] 10.222.0.27 () {60 vars in 1482 bytes} [Thu Jul 14 12:32:23 2022] PUT /identity/v3/projects/dd62f55e2c734a4f8a8c38fa009b157b/users/51bdc8d3518248059037f94067905d18/roles/56f2092f7f5e42b09679dec166d174c5 => generated 0 bytes in 93 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:32:23.506118 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af707950-6165-4c7d-afe6-884de28c0da6 None None] REQUEST_METHOD: `POST` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.506406 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af707950-6165-4c7d-afe6-884de28c0da6 None None] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.506680 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af707950-6165-4c7d-afe6-884de28c0da6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.533161 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.auth.core [None req-af707950-6165-4c7d-afe6-884de28c0da6 None None] MFA Rules not processed for user `51bdc8d3518248059037f94067905d18`. Rule list: `[]` (Enabled: `True`). {{(pid=80488) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:32:23.534421 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-af707950-6165-4c7d-afe6-884de28c0da6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:23.598719 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1436/2902] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:32:23 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:32:23.609296 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:23.610996 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:23.636959 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:23.671956 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a316861f-443b-48c5-b684-a88524f8f14d admin admin] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:23.675595 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5c166d7-cf77-42fe-99be-d3763dc08ede service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e114b292a7f544b0b24016199edcef25', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:23.676230 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c166d7-cf77-42fe-99be-d3763dc08ede service cinder] REQUEST_METHOD: `GET` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:23.676510 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c166d7-cf77-42fe-99be-d3763dc08ede service cinder] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:23.676782 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5c166d7-cf77-42fe-99be-d3763dc08ede service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:23.684883 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c166d7-cf77-42fe-99be-d3763dc08ede service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:23.686571 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5c166d7-cf77-42fe-99be-d3763dc08ede service cinder] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:23.695604 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1467/2903] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:32:23 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:32:33.595805 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:33.739420 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=17 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:34.940057 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af707950-6165-4c7d-afe6-884de28c0da6 None None] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:35.002727 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af707950-6165-4c7d-afe6-884de28c0da6 None None] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:35.007184 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:35.008045 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:35.008324 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:35.008596 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] PATH_INFO: `/v3/users/51bdc8d3518248059037f94067905d18` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:35.009778 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=51bdc8d3518248059037f94067905d18)` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:35.011692 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:35.166326 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '51bdc8d3518248059037f94067905d18'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:35.232663 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '51bdc8d3518248059037f94067905d18'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:35.233272 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '51bdc8d3518248059037f94067905d18'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:35.240593 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '51bdc8d3518248059037f94067905d18'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:35.244141 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.notifications [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '51bdc8d3518248059037f94067905d18'} {{(pid=80488) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:35.258200 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1437/2904] 10.222.0.27 () {58 vars in 1223 bytes} [Thu Jul 14 12:32:34 2022] DELETE /identity/v3/users/51bdc8d3518248059037f94067905d18 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:32:35.262804 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] REQUEST_METHOD: `POST` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:35.263350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:35.263350 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:35.280308 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.auth.core [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] MFA Rules not processed for user `984129f5bec74f43b4a666f392319525`. Rule list: `[]` (Enabled: `True`). {{(pid=80487) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 14 12:32:35.281501 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:35.338623 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1468/2905] 10.222.0.27 () {58 vars in 957 bytes} [Thu Jul 14 12:32:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 14 12:32:35.348173 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:35.350189 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:35.407077 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cefd7f-740e-4292-94e7-da41394432b0 admin admin] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:35.410802 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a4fff90ae5434b58932f468ca9c827f3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:35.411450 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:35.411733 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:35.412048 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:35.421998 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:35.423496 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:35.433021 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1438/2906] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:32:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:32:35.880267 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] Authenticating user token {{(pid=80487) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:32:35.882268 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.fernet_utils [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80487) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:32:35.896520 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d116f9-a3fa-47f7-b449-8d0f464ceaaa None None] Validating token access rules against request {{(pid=80487) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:32:35.901561 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '984129f5bec74f43b4a666f392319525', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1074c03ef1614e229d46654abff5220e', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80487) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:32:35.903425 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:32:35.903713 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] SCRIPT_NAME: `/identity` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:32:35.903988 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] PATH_INFO: `/v3/projects/dd62f55e2c734a4f8a8c38fa009b157b` {{(pid=80487) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:32:35.905185 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dd62f55e2c734a4f8a8c38fa009b157b)` {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:32:35.907203 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] RBAC: Authorization granted {{(pid=80487) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:32:36.059494 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dd62f55e2c734a4f8a8c38fa009b157b', 'request_id': 'req-3bbca4ab-d764-4954-b008-ea30ab546170'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:36.127461 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dd62f55e2c734a4f8a8c38fa009b157b', 'request_id': 'req-3bbca4ab-d764-4954-b008-ea30ab546170'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:36.134439 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dd62f55e2c734a4f8a8c38fa009b157b', 'request_id': 'req-3bbca4ab-d764-4954-b008-ea30ab546170'} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:36.144726 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] The token cache is being invalidate because project dd62f55e2c734a4f8a8c38fa009b157b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80487) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 14 12:32:36.145013 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:36.145644 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG keystone.notifications [None req-3bbca4ab-d764-4954-b008-ea30ab546170 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80487) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 14 12:32:36.147236 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: [pid: 80487|app: 0|req: 1469/2907] 10.222.0.27 () {58 vars in 1232 bytes} [Thu Jul 14 12:32:35 2022] DELETE /identity/v3/projects/dd62f55e2c734a4f8a8c38fa009b157b => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 14 12:32:45.428818 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:32:46.141768 dv-cinder-848900-3-iscsi devstack@keystone.service[80487]: DEBUG dbcounter [-] [80487] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80487) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 14 12:37:33.850653 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] Authenticating user token {{(pid=80488) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 14 12:37:33.852093 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:37:33.989687 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a79bbf-3527-43bd-9843-c4079629479b service neutron] Validating token access rules against request {{(pid=80488) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 14 12:37:34.048485 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56303a1f-a704-4a5f-8f37-30f9a6cfa02e service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'a9c50ae1ba494c20b6133a96dfba21b3', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'aa1d22dbd90a47539c4e594aee69a482', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'service', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80488) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 14 12:37:34.049186 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56303a1f-a704-4a5f-8f37-30f9a6cfa02e service placement] REQUEST_METHOD: `GET` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 14 12:37:34.049463 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56303a1f-a704-4a5f-8f37-30f9a6cfa02e service placement] SCRIPT_NAME: `/identity` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 14 12:37:34.049738 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56303a1f-a704-4a5f-8f37-30f9a6cfa02e service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80488) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 14 12:37:34.052201 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.fernet_utils [None req-56303a1f-a704-4a5f-8f37-30f9a6cfa02e service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80488) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 14 12:37:34.063584 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56303a1f-a704-4a5f-8f37-30f9a6cfa02e service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 14 12:37:34.065146 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56303a1f-a704-4a5f-8f37-30f9a6cfa02e service placement] RBAC: Authorization granted {{(pid=80488) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 14 12:37:34.075046 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: [pid: 80488|app: 0|req: 1439/2908] 10.222.0.27 () {60 vars in 1365 bytes} [Thu Jul 14 12:37:33 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 225 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 14 12:37:44.070670 dv-cinder-848900-3-iscsi devstack@keystone.service[80488]: DEBUG dbcounter [-] [80488] Writing DB stats keystone:SELECT=89 {{(pid=80488) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}